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Preface 


The  research  which  has  resulted  in  this  thesis  was 
accomplished  to  assist  those  assigned  the  task  of  analyzing, 
or  managing  the  analysis  of.  Test  and  Evaluation  data.  *The 
guide  was  written  as  an  appendix  so  that  it  can  be  easily 
extracted  from  the  thesis.  The  guide  was  written  so  that 
analysts  with  very  little  mathematical  background  could  use 
it  effectively.  The  success  of  the  thesis  will  be  determined 
by  how  easy  the  guide  is  to  use  effectively.  The  orientation 
of  the  thesis  also  resulted  in  the  inclusion  of  much  intro¬ 
ductory  material,  which  will  not  be  useful  to  all  readers. 

In  the  attempt  to  write  a  very  precise  and  technical  guide 
in  an  unsophisticated  language  there  are  undoubtedly  some 
innaccuracies,  but  that  is  the  challenge  of  this  kind  of 
research. 

I  wish  to  express  my  gratitude  to  Lieutenant  Colonel 
Ivy  Cook,  my  advisor,  and  Lieutenant  Colonel  Richard  Kulp, 
my  reader.  Both  were  very  helpful  in  all  phases  of  the  re¬ 
search  and  the  quality  of  this  thesis  is  a  reflection  of 
their  guidance  and  patience. 
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This  research  effort  was  directed  toward  developing  a 
practical,  easy-to-use  guide  for  statistical  analysis  of 
Test  and  Evaluation  data.  Ease  of  use  and  completeness  have 
both  been  stressed  in  writing  this  guide.  The  guide  is 

V 

written  for  users  with  little  mathematical  background,  how¬ 
ever  should  be  useful  to  all  Test  and  Evaluation  analysts 
and  managers.  The  guide  includes  a  discussion  of  the  basic 
statistical  terminology  as  well  as  a  step-by-step  analysis 
procedure.  The  analysis  procedure  is  incorporated  into  a 
flow  chart  and  includes  discussions  and  examples  of  indivi¬ 
dual  statistical  tests.  The  statistical  tests  include 
parametric  and  nonparametric  tests  which  can  be  applied  to 
most  test  data  types. 
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A  GUIDE  FOR  THE  APPLICATION  OF 
STATISTICAL  ANALYSIS  METHODS  TO 
OPERATIONAL  TEST  AND  EVALUATION  DATA 


I.  Introduction 

Department  of  Defense  (DOD)  and  Air  Force  directives  and 
regulations  require  that  the  evaluation  of  weapon  systems  be 
carried  out  in  such  a  way  that  subjective  judgement  is  mini¬ 
mized  and  quantitative  analysis  is  maximized  (Ref  26:2).  In 

,  .  .  .  .  .  1 

accomplishing  the  quantitative  analysis  for  Air  Force  test 
and  evaluations ,  statistical  methods  have  been  the  most  widely 
and  predominantly  used.  To  effectively  evaluate  a  weapon 
system,  the  Air  Force  analyst  must  bring  to  his  task  suffic¬ 
ient  operational  experience  so  that  the  proper  issues  are 
addressed  and  a  working  knowledge  of  statistical  analysis  so 
that  the  quantitative  data  can  be  analyzed  correctly.  Unfor¬ 
tunate^  many  test  analysts  are  assigned  to  tests  based  solely 
on  their  operational  experience,  with  little  or  no  training 
with  the  analytical  tools  which  are  needed  to  accomplish 
their  task.  However,  no  primer  or  guide  exists  that  would 
enable  those  with  minimal  mathematical  background  to  perform 
the  required  statistical  analysis  satisfactorily. 

Objective 

The  objective  of  the  thesis  is  to  provide  a  guide  that 


1 


\ 


I 


will  assist  test  analysts  in  the  analysis  and  evaluation  of 

Air  Force  weapon  systems.  The  guide  should  be  most  useful  to  -  .  • 

analysts  with  little  or  no  math  .atical  expertise,  but  may  be 

% 

used  as  a  quick  reference  by  all  analysts  and  managers  in  the 

-  y 

field.  The  guide  is  general  in  nature  so  that  the  methods 

‘  * 

can  be  used  in  broad  applications,  however,  not  so  general  ^ 

that  it  is  not  applicable  to  specific  analyses.  The  guide 
was  written  in  a  step-by-step  format  so  that  a  minimum  of 
mathematical  background  is  required  of  the  user.  The  methods 
recommended  in  the  guide  are  well  known,  well  documented  in 
referenced  sources  of  various  mathematical  skill  levels, 
easily  applied,  and  readily  available  in  computer  statistical 
packages. 

Background  > 

As  an  entry  level  analyst  (AFSC  2681) ,  with  no  analytical 
or  operational  experience,  I  was  assigned  to  the  Tactical  Air 
Command  F-16  Multinational  Operational  Test  and  Evaluation 
Analysis  Branch  at  Hill  AFB,  Utah  and  in  Europe.  Fully  half 
of  the  operational  analysts  had  no  previous  test  experience 
and  came  from  various  operational  and  academic  backgrounds. 

In  the  search  for  guides  and  manuals  which  would  aid  us  in 
defining  our  responsibilities  and  the  purpose  of  our  job,  I 
found  only  information  which  was  too  general  and  no  informa¬ 
tion  which  provided  guidance  in  all  phases  of  the  analysis 
of  test  data. 
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Conversations  with  other  test  analysts  and  my  own  ex¬ 
perience  highlighted  the  need  for  a  guide  which  the  inexperi¬ 
enced  analyst  could  consult  as  *»n  introduction  to  Operational 
Test  and  Evaluation  (OT&E)  and  later  as  a  reference  for. de¬ 
signing  and  carrying  out  test  data  analysis.  The  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  was  contacted  to  determine 
whether  or  not  they  viewed  a  guide  of  this  sort  as  a  worth¬ 
while  project  which  they  would  support  as  a  thesis  sponsor. 
AFTEC  has  sponsored  the  research  in  hopes  that  it  can  be  pub¬ 
lished  as  an  analyst's  guide  to  statistical  data  analysis. 

Scope 

The  coverage  of-  statistical  analysis  methods  is  limited 
to  those  which  have  the  most  general  applicability  to  opera¬ 
tional  testing.  The  methods  recommended  should  provide  at 
least  one  method  of  analysis  for  each  type  of  objective  and 
quantitative  data  encountered.  Parametric  and  nonparametric 
statistical  tests  are  used  in  the  guide  so  that  data  samples 
which  fail  tests  of  key  assumptions  or  have  small  sample 
sizes  can  be  analyzed. 

Limitations 

The  guide  has  been  developed  based  on  the  review  of 
twenty-six  OT&E  Final  Reports  and  Test  Plans.  Thus  some  meas¬ 
ures  of  effectiveness  (MOE)  may  not  have  been  identified  or 
anticipated,  so  that  no  method  is  recommended  here  or  a 
better  method  may  be  available. 
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The  guide  is  by  no  means  self-contained,  that  is,  the 
analyst  must  have  access  to  probability  distribution  tables, 
statistical  references  and  computer  support  in  some  cases. 

The  guide  was  purposefully  written  with  a  low  degree  of 
mathematical  complexity,  and  some  advanced  concepts  were 
omitted  or  only  referenced.  It  was  felt  that  simplicity  and 
generality  contribute  most  to  the  guide's  utility  and  this 
philosophy  has  been  the  over-riding  determinant  of  what  has 
been  included. 

There  is  no  coverage  given  to  the  treatment  of  qualita¬ 
tive  or  subjective  survey  data.  While  this  area  may  have 
substantial  contributions  for  OT&E,  the  analytical  methods 
are  secondary  to  the  proper  development  of  the  survey.  Cur¬ 
rently  there  is  no  guide  available  for  developing  a  subjective 
analysis. 


[?>, 
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Table  I 


Test  Reports  Reviewed 

AIM-9M  Initial  Operational  Test  and  Evaluation,  Final  Report 
AGM-88  (HARM)  Air  Force  Preliminary  Evaluation  Report 
AIM-7F  Follow-on  Operational  Test  and  Evaluation,  Final  Report 

Simplified  Processing  Station  Phase  I  Follow-on  Operational 
Test  and  Evaluation,  Final  Report 

Advanced  Medium  STOL  Transport  Analysis  Methodology 
C-141B  Initial  Operational  Test  and  Evaluation, 

Test  Plan  and  Final  Report 

A— 10  Initial  Operational  Test  and  Evaluation,  Test  Plan  and 
Final  Report 

A— 10  Follow-on  Operational  Test  and  Evaluation,  Final  Report 

F— 111  Initial  Operational  Test  and  Evaluation,  Final  Report 

EF-111  Initial  Operational  Test  and  Evaluation,  Test  Plan 
and  Final  Report 

F-15  Developmental  Test  and  Evaluation  Final  Report 

F-15  Initial  Operational  Test  and  Evaluation,  Final  Report 

F-15  Follow-on  Operational  Test  and  Evaluation,  Final  Report 

F-16  Initial  Operational  Test  and  Evaluation,  Final  Report 

F-16  European  Test  and  Evaluation,  Test  Plan  and  Final  Report 

F-16  Phase  I  Follow-on  Operational  Test  and  Evaluation,  Test 
Plan,  Detailed  Analysis  Products,  and  Final  Report 

F-16  Phase  II  Follow-on  Operational  Test  and  Evaluation, 

Test  Plan,  Detailed  Analysis  Products,  and  Final  Report 

F-16  Phase  II  Follow-on  Operational  Test  and  Evaluation 
Suitability,  Test  Plan  and  Final  Report 
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II .  Methodology 


Initial  research  was  conducted  to  determine  if  the  pro¬ 
posed  thesis  had  already  been  accomplished  in  some  form.  A 
search  of  Air  Force  literature  and  MITRE  Corporation,  DTIC, 
and  DLSIE  bibliography  searches  revealed  a  lack  of  general 
OT&E  analysis  methodologies.  The  Air  Force  (AFTEC) ,  Army 
(OTEA)  and  Navy  (OPTEVFOR)  operational  test  centers  were  all 
contacted  to  determine  whether  they  had  a  guide  for  the  test 
analyst.  A  Navy  publication  (Ref  23)  was  the  only  document 
discovered  which  was  written  expressly  for  an  analyst's  use. 
This  document.  Analyst's  Notebook,  does  not  provide  a  step- 
by-step  procedure,  is  of  a  very  specific  nature,  and  does 
not  cover  the  entire  analysis  effort.  All  three  services' 
operational  testing  centers  expressed  interest  in  this  re¬ 
search  as  something  which  is  necessary  and  unavailable. 

The  guide  itself  (Appendix  A)  has  been  written  with 
three  distinct  objectives  in  minds 

1)  an  introduction  to  the  purpose,  nature  and  structure 
of  Air  Force  OT&E 

2)  an  introduction  to  statistical  terms  and  methods 
which  are  commonly  used  in  Air  Force  OT&E 

3)  a  step-by-step  guide  for  the  analysis  of  OT&E  data 
which  enables  and  encourages  the  proper  handling  of 
design,  analysis  and  interpretation  by  the  analyst. 

The  first  two  objectives  were  accomplished  after 


reviewing  current  Department  of  Defense  and  Air  Force  direc¬ 
tives  and  regulations,  related  books  and  technical  reports 
as  well  as  a  review  of  statisti  1  literature  and  the  Final 
Reports  from  17  recent  Air  Force  tests  (Table  I) . 

The  step-by-step  method  for  the  guide  was  developed  into 
a  flow  chart  to  minimize  the  bulk  of  the  guide  and  to  provide 
a  better  graphical  representation  of  the  analysis  procedure. 
Where  possible  the  required  calculations  are  included  in  the 
flow  chart;  however,  due  to  the  complexity  of  some  steps, 
individual  descriptions  are  also  included  in  the  guide  with 
references,  examples,  and  the  correct  program  from  both  SPSS 
and  BMD  systems.  The  analysis  procedure  outlined  in  the  guide 
includes  the  testing  of  the  key  assumptions  of  the  tests  prior 
to  their  use  with  an  alternative  path  if  an  assumption  is  re¬ 
jected.  With  the  guide  in  this  format,  it  is  hoped  that  the 
important  intermediate  steps  in  the  analysis  will  not  be 
overlooked. 

The  flow  chart  was  developed  using  some  non-standard 
statistical  notation  in  the  interest  of  generality  and  to 
minimize  the  complexity  of  the  flow  chart  itself.  The  flow 
chart  serves  the  dual  purpose  of  determining  the  type  test 
and  sample  size  in  the  design  phase  as  well  as  determining 
the  proper  test  in  the  analysis  phase. 

In  addition  to  the  flow  chart,  descriptions  and  examples 
of  the  statistical  tests,  the  guide  also  provides  the  analyst 
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with  suggestions  for  reporting  the  results  of  the  analysis, 
contacts  within  the  Air  Force  for  advice  on  all  phases  and  a 
bibliography  which  lists  the  references  in  increasing  order 
of  mathematical  complexity. 
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Ill .  Results  and  Conclusions 


The  review  of  seventeen  USAF  Test  Reports  not  only  pro¬ 
vided  the  basis  for  the  scope  and  coverage  of  the  guide 
(Appendix  A),  but  also  highlighted  the  need  for  such  a  guide. 
In  all  of  the  reports  there  was  very  little  evidence  of  any 
statistical  analysis  being  done  at  all.  In  the  few  reports 
which  did  report  the  results  of  the  statistical  analysis, 
the  treatment  indicated  little  understanding  of  the  methods 
used.  This  problem,  too  little  use  of  sound  statistical 
analysis,  may  be  the  result  of  too  few  qualified  test  analysts 
or  the  reluctance  of  test  management  to  include  analyses 
which  they  feel  are  tbo  complex  or  won't  be  understood.  What¬ 
ever  the  case,  a  standardized,  easy-to-use,  easy-to-understand 
guide  using  basic  statistical  analysis  methods  could  be  used 
by  both  analyst  and  test  manager  alike  to  overcome  the  pro¬ 
blem. 

That  easy-to-use  guide  is  the  result  of  this  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  sponsored  research.  The 
guide  includes:  a  discussion  of  Air  Force  Test  and  Evaluation 
(T&E) ,  a  discussion  of  basic  statistical  terms  and  their  re¬ 
lation  to  the  test  environment  as  well  as  a  step-by-step 
guide  for  using  statistical  analysis  on  T&E  data.  Much  of 
the  background  material  could  not  be  omitted  or  abbreviated 
if  the  guide  is  to  be  useful  to  all  the  prospective  users; 
however  the  actual  analysis  section  was  reduced,  by  the  use 
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of  a  flow  chart.  The  flow  chart  is  helpful  in  that  it  pre¬ 
sents  all  of  the  steps  required  in  an  analysis,  some  of  which  - 
are  often  neglected.  The  flow  chart  will  ensure  that  alterna¬ 
tive  methods  are  at  least  discovered  if  not  used  and  should 
result  in  better  analyses  if  followed  correctly. 

The  Air  Force  T&E  community  is  in  need  of  a  standardized 
method  for  analyzing  test  data;  the  attached  guide  should  be 
a  good  beginning  toward  this  end.  Further  work  in  the  area 
of  subjective  evaluation,  and  other  measures  of  effectiveness 
could  be  accomplished  as  well  as  the  study  of  the  applicability 
of  specific  tests  to  T&E  data.  Test  and  Evaluation  is  a  dif¬ 
ficult  area  in  which  to  apply  statistical  analysis  because 
the  objective  of  the  test  must  be  weighed  against  the  cost 
of  testing;  however  it  is  also  an  area  where  good  statistical 
analysis  is  needed.  This  guide  may  form  the. basis  for  an 
improvement  in  current  T&E  analysis  practices. 

The  Test  Plans  and  Test  Reports  which  were  reviewed  and 
conversations  with  Air  Force,  Navy  and  Army  test  managers  and 
analysts  indicate  that  the  conduct  of  Operational  Tests  does 
not  satisfy  the  intent  of  the  pertinent  regulations  concerning 
quantitative  analysis  of  test  data.  The  Operational  Testing 
environment  does  not  allow  for  significant  input  from  the 
analysis  function  concerning  experimental  design  or  test  con¬ 
duct.  This  problem  can  not  be  corrected  without  significant 
changes  in  the  role  of  test  analysts  and  the  attitudes  of 


test  managers.  However  the  attitude  of  test  managers  may  be 
altared  by  their  education  in  statistical  analysis  methods. 
This  guide  may  be  as  useful ,  to  the  analysts,  in  their  bosses 
hands  as  it  will  be  in  their  own. 

To  make  this  sort  of  step-by-step  guide  even  more  useful 
to  test  analysts  it  may  be  practical  to  develop  an  interactive 
software  package  which  could  be  used  to  perform  data  analysis 
without  accessing  other  statistical  packages,  e.g.  BMD  and 
SPSS,  or  performing  any  hand  calculations.  The  development 
of  such  an  analysis  tool  for  mini -computers  or  larger  systems 
would  go  a  long  way  toward  standardizing  and  improving  current 
test  data  analysis. 

The  current  state  of  USAF  Operational  Test  data  analysis 
is  such  that  improvements  must  be  made  in  order  to: 

1)  improve  the  overall  quality  of  the  analyses, 

2)  standardize  analysis  procedures,  and 

3)  develop  a  more  knowledgeable  and  understanding  test 
management,  concerning  test  data  analysis. 

These  changes  should  result  in  the  development  of  better 
testing  techniques  and  more  confidence  in  test  results.  A 
guide  or  similar  analysis  tool  is  the  first  step  in  obtaining 
these  improvements. 

Recommend  that  this  step-by-step  guide  be  the  basis  for 
a  standardized  analysis  procedure  to  be  instituted  by  AFTEC 
and  that  further  research  be  directed  towards  an  interactive 
computer  analysis  system. 
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APPENDIX  a 


A  GUIDE  FOR  THE  STATISTICAL 
ANALYSIS  OF  OPERATIONAL  TEST  DATA 
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I. 


INTRODUCTION 


The  analysis  of  Operational  Test  and  Evaluation  (OT&E) 
data  requires  the  mixing  of  operational  knowledge  and  statis- 
tical  theory.  Applying  theory  to  any  practical  problem  is 
usually  difficult;  applying  statistical  theory  to  Test  and 
Evaluation  (T&E)  analysis  is  no  exception.  The  guide  presents 
a  step-by-step  procedure  which  can  be  used  for  OT&E  data  anal¬ 
ysis  by  the  analyst  and  manager.  It  may  be  of  most  help  to 
the  new  OT&E  analyst  who  does  not  have  an  extensive  analytical 
background.  However,  as  a  quick  reference  to  the  most  common 
statistical  tests,  the  guide  can  be  useful  to  anyone  involved 
in  T&E. 

The  purpose  of  the  guide  is  to  present  a  brief,  simple 
and  complete  plan  of  attack  for  analyzing  the  most  common 
OT&E  objectives.  In  the  interest  of  brevity  and  simplicity 
it  will  be  presented  without  much  statistical  detail  but  will 
concentrate  on  maximizing  its  usefulness.  The  guide  is  com¬ 
plete  in  the  sense  that  it  presents  at  least  one  method  for 
dealing  with  each  of  the  areas  to  be  analyzed. 

The  guide  is  based  on  recent  Final  Reports  and  Test 
Plans  from  the  T&E  of  current  USAF  weapon  systems.  In  the 
reports  reviewed,  three  basic  types  of  measures  of  effective¬ 
ness  (MOE)  were  identified.  Other  MOE's  may  be  available, 
but  since  these  three  types  will  provide  a  solid  base,  the 
guide  is  limited  to  them:  proportions,  averages,  and  measures 
of  dispersion. 
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II. 


THE  PHILOSOPHY  OF  TESTING 


Military  Test  and  Evaluation  has  developed  as  a  major 

step  in  the  systems  acquisition  process  "...  to  identify, 

*• 

assess  and  reduce  the  acquisition  risk,  to  evaluate  opera¬ 
tional  effectiveness  and  operational  suitability,  and  to 
identify  any  deficiencies  in  the  systems."  (Ref  26:2)  Air 
Force  T&E  is  divided  into  two  major  categories:  OT&E  and 
Development  Test  and  Evaluation  (DT&E) .  Analysis  and  test 
conduct  are  very  similar  for  both,  however,  they  have  separate 
purposes.  While  this  guide  concentrates  on  OT&E,  many  of  the 
applications  can  be  made  to  DT&E  directly. 

Operational  Test  and  Evaluation  is  that  testing  "... 
conducted  to  estimate  a  system's  operational  effectiveness 
and  operational  suitability,  identify  needed  modification, 
and  provide  information  on  tactics,  doctrine,  organization 
and  personnel  requirements."  (Ref  27:3) 

Development  Test  and  Evaluation  is  that  "...  testing 
and  evaluation  used  to  measure  progress,  verify  accomplish¬ 
ment  of  development  objectives,  and  to  determine  if  theories, 
techniques,  and  material  are  practicable;  and  if  systems  or 
items  under  development  are  technically  sound,  reliable, 
safe,  and  satisfy  specif ications,"  (Ref  26:30) 

OT&E  differs  from  DT&E  in  that  OT&E  tests  systems  which 
have  progressed  through  the  procurement  process  to  the  point 
where  the  system  can  actually  function  in  an  operational 
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environment.  OT&E  is  the  first  opportunity  for  the  using 
command  to  critically  evaluate  the  system.  This  can  often 
lead  to  an  adversary  relationship  with  the  engineering  and 
procurement  functions  which  execute  the  development  phages. 

The  importance  of  OT&E  to  an  effective  procurement  pro¬ 
cess  cannot  be  overstated.  In  August  1967  the  President's 
Scientific  Advisory  Committee  (military  aircraft  panel) 
stated  "...  we  believe  most  of  our  previous  failures  to  be 
prepared  for  wars  we  fight  would  have  been  thoroughly  exposed 
had  an  adequate  program  of  testing  and  evaluation  existed." 
(Ref  31:8)  The  necessity  for  honest,  effective  testing  has 
become  clearer  in  recent  years  due  to  the  ever  increasing 
cost  of  military  weapon  systems  and  the  ever  increasing 
threat . 

The  analyst's  responsibility  in  the  OT&E  environment  is 
to  carry  the  test  beyond  the  realm  of  subjective  judgment, 
by  providing  the  methods  and  techniques  for  quantifying  and 
analyzing  system  performance.  "Quantitative  data  must  be 
used,  to  the  maximum  extent  practical,  to  show  that  the  major 
objectives  have  been  met.  Subjective  judgment,  relative  to 
system  performance,  must  be  minimized."  (Ref  26:2) 

The  analyst  is  responsible  to  the  Test  Director  for  the 
design  of  the  test,  which  should  produce  quantifiable  results 
that  can  withstand  the  scrutiny  of  operational  and  scientific 
experts.  Additionally,  the  analyst  is  responsible  for  analy¬ 
zing  the  data  collected  during  the  test.  In  carrying  out 
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the  analysis  function,  the  analyst  must  above  all  else  main¬ 
tain  impartiality.  Interpretation  of  the  results  of  any  anal¬ 
ysis  is  often  done  by  someone  other  than  the  analyst.  To 
enable  others  to  do  this  correctly,  the  analyst  should  pro¬ 
vide  an  analysis  which  is  objective  and  repeatable.  The 
statement  of  all  the  assumptions  is  required  as  well  as  any 
other  possible  limitations.  Basically,  the  analyst's  func¬ 
tion  is  to  translate  a  complex  question  into  a  number  of 
options  which  have  been  enumerated  and  can  be  compared.  The 
analyst  will  be  responsible  for  presenting  an  analysis  that 
aids  the  decision  maker?  however,  final  responsibility  for 
any  decision  always  rests  with  the  decision  maker  alone. 

The  budget  constraint  will  most  often  be  felt  because  of 
its  affect  upon  the  amount  of  data  which  can  be  collected. 
Whether  testing  aircraft  or  munitions,  there  is  usually  a 
substantial  expenditure  associated  with  each  sortie  or  trial. 

A  limited  budget  reduces  either  the  number  of  objectives 
which  can  be  answered  or  the  number  of  trials  devoted  to  a 
single  objective.  The  budget  constraint  will  almost  always 
be  in  effect,  so  that  the  decision  must  be  made  between  lim¬ 
iting  the  objectives  or  limiting  replications.  The  objectives 
of  any  test  should  be  scoped  down  so  that  each  area  is  covered 
adequately. 

The  lack  of  measurement  Instrumentation  is  often  a  con¬ 
straint  due  to  the  cost  of  making  it  available  or  because  of 
integration  time  requirements.  To  perform  a  test  with 
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quantifiable  results  will  often  require  special  equipment, 
like  video  tape  recorders  (VTR)  or  time,  space,  position 
instrumentation  (TSPI) ,  to  provide  the  necessary  data.  A 
lack  of  test  instrumentation  will  result  in  less  accurate 
quantitative  data;  however,  instrumented  test  systems  alone 
are  not  always  enough.  Quite  often  support  systems  or  threat 
simulators  must  also  be  instrumented  to  make  any  of  the  data 
useable.  Test  Instrumentation  must  be  a  well  planned  and 
coordinated  effort  to  be  effective,  but  the  rewards  are  often 
worth  the  additional  effort. 

"OT&E  should  be  accomplished  in  an  environment  as  opera¬ 
tionally  realistic  as  possible."  (Ref  27:3).  Often  the 
desire  to  test  under  operationally  realistic  conditions  is 
in  direct  conflict  with  the  collection  of  quantitative  data. 

In  some  cases,  test  instrumentation  cannot  function  in  real¬ 
istic  conditions  and  the  need  for  realism  must  be  weighed 
against  the  need  for  quantifiable  data. 

Perhaps  a  more  basic  question  than  "What  are  the  condi¬ 
tions  under  which  the  data  is  collected?"  or  "What  data  should 
be  collected?"  is  the  question  "How  accurate  is  the  data?" 

Data  accuracy  can  be  affected  by  test  design  but  more  often 
is  affected  by  the  data  collection  process.  Usually  an 
analyst’s  responsibility,  data  collection  must  be  performed 
in  the  most  accurate  way.  Analyzing  OT&E  data  is  very  diffi¬ 
cult  because  of  the  operational  influences;  however,  the  job 
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becomes  arly  impossible  to  do  well  if  excessive  error  is 
introduced  at  the  point  where  data  is  collected,  reviewed, 
or  processed. 


The  main  tools  for  data  collection  are  test  instrumenta¬ 
tion  and  manual  data  forms.  Before  any  data  is  collected, 
the  objective (s)  for  which  it  will  be  used  should  be  identi¬ 
fied  and  an  analysis  technique  should  be  specified.  Data 
collection  forms  should  be  designed  to  minimize  the  possi¬ 
bility  of  human  error  by  limiting  their  content  to  essential 
items  and  by  proper  formatting.  Simple  forms  will  reduce  the 
possibility  of  incorrect  data  collection.  The  forms  should 
be  self-explanatory  and  easy  to  follow.  Proper  design  of 
forms  can  allow  the  operators  to  complete  forms  and  eliminate 
the  need  for  dedicated  data  collectors  to  conduct  interviews. 

Subjective  judgments  contribute  much  to  current  OT&E 
final  reports.  The  reasons  for  this  include  the  lack  of 
adequate  instrumentation,  unquantif iable  test  objectives  and 
the  lack  of  accurate  quantitative  data.  The  use  of  subjec¬ 
tive  data  analysis  has  been  neglected  for  the  most  part  in 
USAF  tests.  Subjective  survey  data  can  never  replace  quan¬ 
tifiable  results ‘  however,  when  analyzed  with  the  proper 
analysis  techniques,  it  can  be  used  to  lend  support  to  the 
subjective  judgment. 


III.  STATISTICS  AND  TESTING 


The  use  of  statistical  analysis  methods  is  not  required 

in  Air  Force  testing;  however,  when  compared  with  other  tech- 

• 

niques,  such  as  decision  analysis  or  cost-benefit  analysis, 
it  becomes  clear  that  statistical  analysis  has  many  advan¬ 
tages  .  Statistical  methods  are  familiar  to  many  decision 
makers.  They  are  well  documented,  easy  to  interpret  and  easy 
to  apply.  The  majority  of  OT&E  analysis  has  been  done  using 
statistical  methods  and  there  is  a  lot  of  information  in 
technical  reports  on  specific  applications. 

To  effectively  use  statistical  analysis  a  "correct" 
technique  must  be  found;  this  technique  must  be  applied  cor¬ 
rectly;  and  the  results  must  be  interpreted  correctly.  OT&E 
is  not  an  easy  area  to  apply  statistical  methods  because  of 
the  presence  of  so  many  uncontrollable  and  unmeasureable 
variables.  Creativity  and  caution  must  both  be  applied  in 
equal  amounts  in  all  phases  of  the  analysis.  Creativity  - 
because  many  times  OT&E  analysis  is  completely  original,  has 
not  been  documented,  or  at  the  very  least,  is  a  new  applica¬ 
tion  of  an  old  method.  Caution  must  be  exercised  because 
OT&E  data  analysis  is  more  than  an  art.  It  is  also  a  science, 
with  some  strict  rules  which  cannot  be  broken  if  the  analysis 
is  to  withstand  inspection  by  professional  statisticians. 

The  objective  of  the  analysis  must  be  translated  from 

i 

the  language  of  the  Test  Plan  into  a  statistical  hypothesis. 
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This  hypo-thesis  (H0)  ,  called  the  null  hypothesis,  made  in  the 
form  of  a  mathematical  formula,  can  be  tested,  and,  if  re¬ 
jected,  an  alternate  hypothesis  (Ha)  which  is  the  exact  com¬ 
plement  can  be  accepted.  For  example: 

H  :  Circular  Error  Probable  (CEP)  >  military  specifica- 
o 

t ion  (e.g.  100') 

Ha:  CEP  <  military  specification. 

If  the  stated  hypothesis  (Hc)  is  rejected  statistically,  then 
Ha  must  be  accepted. 

Choosing  the  proper  statistical  model  for  the  combina¬ 
tion  of  objective  and  data  characteristics  is  the  hardest 
step  in  the  procedure  and  can  most  affect  the  results.  The 
first  criterion  in  choosing  a  model  is  cata  dependent,  but 
subject  to  the  analyst's  decision.  Models  are  of  tvo  basic 
types:  parametric  and  nonparametric .  The  parametric,  models. 
and  the  associated  tests  are  based  on  relatively  specific 
assumptions  concerning  characteristics  of  the  population  from 
which  the  data  is  drawn.  The  assumptions  include:  measura¬ 
bility  of  the  data,  shape  of  the  population  probability  dis¬ 
tribution  (e.g.,  normal  distribution)  and  the  number  of 
populations  being  sampled.  The  effectiveness  of  the  para¬ 
metric  tests  is  directly  related  to  the  validity  of  these 
assumptions.  However,  some  of  the  tests  still  function 
relatively  well  with  slight  violations  to  the  assumptions 
concerning  the  shape  of  the  sampled  distribution.  This  is 


especially  true  of  tests  of  averages. 


The  nonparametric  tests  are  sometimes  referred  to  as 
distribution-free  because  they  do  not  contain  strict  assump¬ 
tions  concerning  the  probability  distribution  of  the  sampled 
population.  The  nonparametric  tests  do  make  assumptions  of 
measurability  and  population  characteristics,  but  they  are  more 
general  in  nature  and  are  applicable  more  often.  Because  of 
the  relaxed  assumptions,  the  nonparametric  tests  will  not 
identify  statistical  differences  as  readily  as  the  parametric 
tests  (Ref  9:32-33);  however,  this  is  true  only  when  the 
assumptions  of  the  parametric  tests  hold.  In  cases  where 
the  assumptions  of  the  parametric  test  do  not  hold,  the  re- 
suits  of  parametric  tests  are  unpredictable.  Thus  the  accept¬ 
ance  or  rejection  of  a  statistical  hypothesis  may  be  caused 
by  the  failure  of  the  assumptions  rather  than  the  validity 
or  invalidity  of  the  hypothesis  being  tested.  For  further 
discussion  of  nonparametric  statistics  see  (Ref  9:30-34)  and 
(Ref  10:91-93). 

Another  important  use  for  nonparametric  statistics  in 
the  OT&E  environment  is  in  its  applicability  to  qualitative 
data.  Qualitative  or  subjective  data  can  usually  be  collected 
in  terms  of  rankings  or  categories.  The  advantage  here  is 
that  subjective  data  is  inexpensive  to  collect,  usually  re¬ 
sults  in  large  samples  and  can  be  used  to  support  quantitative 
results  to  evaluate  an  unquantif iable  objective.  Multivariate 
analysis  methods,  such  as  multiple  regression  and  discriminant 
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analysis  have  also  been  applied  to  subjective  data  with 
success  outside  of  OT&E.  Any  analysis  of  subjective  data  is 
very  dependent  upon  the  conten  and  format  of  the  collection 
form.  The  design  of  subjective  surveys  is  nearly  a  science 
in  itself  and  should  not  be  undertaken  carelessly. 
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IV.  STATISTICAL  TERMS 


The  application  of  statistical  theory  to  test  data  is 
most  often  done  with  classical  statistical  hypothesis  testing 
and  its  associated  techniques.  A  clearly  stated  objective  is 
the  cornerstone  of  a  good  analysis,  but  in  OT&E,  just  as  im¬ 
portant  is  the  measure  of  effectiveness  (MOE) .  The  MOE  will 
be  used  to  determine  if  the  objective  has  been  met  and,  as 
the  name  implies,  must  be  measurable.  The  scale  of  measure¬ 
ment  can  be: 

1)  Nominal  or  Categorical  -  the  least  preferred  type 
of  data,  e.g.  aircraft  type  (F-4,  F-16) ;  air  combat  outcome 
(win,  loss,  draw) . 

2)  ‘  Ordinal  or  Ranked  -  e.g.  altitude  (low,  medium,  high), 
subjective  evaluation  (inferior  to  current  systems,  as  good 

as  current  systems,  superior  to  current  systems). 

3)  Interval  -  the  distance  between  observable  values  is 
consistent,  however  the  zero  is  not  an  absolute  value,  e.g. 
temperature  in  F°  or  C°;  IQ  or  SAT  scores. 

4)  Ratio  -  the  data  is  interval  and  an  absolute  2ero 
exists  which  makes  ratios  comparable,  e.g.  distance  (feet), 
probability  of  kill,  normal  acceleration  (g's),  velocity 
(knots),  temperature  in  °K. 

A  further  distinction  in  the  measurability  of  data  which 
concerns  only  ratio  and  interval , measures  is  the  notion  of 
discrete  and  continuous  data.  Continuous  data  should  be  able 
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to  take  on  all  values  between  the  extremes  whereas  discrete 
data  can  take  on  only  a  finite  or  countably  infinite,  e.g. 
all  integers,  number  of  values  For  further  discussion  of 
the  scales  of  measurement  see  {Ref  9:21-30). 

How  well  an  objective  can  be  analyzed  will  be  determined 
by  the  characteristics  of  the  MOE.  The  MOE  must  be  readily 
measured  in  an  unambiguous  way  and  should  relate  directly  to 
the  objective.  The  more  that  is  known  about  the  probabilistic 
nature  of  the  MOE,  the  easier  it  will  be  to  determine  the  cor¬ 
rect  analytical  model.  If  nothing  or  little  is  known  about 
the  MOE,  it  will  usually  be  best  to  assume  that  it  is  normally 

distributed,  if  it  is  continuous,  until  shown  to  be  otherwise. 

* 

In  statistical  applications  the  data  that  is  collected 
is  termed  a  sample  and  is  assumed  to  be  a  random  selection  from 
an  underlying  population.  This  population  is  often  assumed 
to  have  a  specific  statistical  distribution,  e.g.  normal  , 
which  is  the  basis  of  the  parametric  tests  in  this  guide. 

The  term  “statistical  confidence"  is  one  that  is  common  to 
nearly  all  applications  and  is  represented  by  the  term  (1-a) 
xl00%.  The  significance  level,  a,  is  the  probability  of  re¬ 
jecting  a  stated  hypothesis  (Hq)  when  it  is  true,  called  a 
type  I  error.  Confidence  is  important  when  HQ  has  been  re¬ 
jected,  as  it  gives  an  indication  of  how  likely  it  is  that 
you  have  drawn  the  correct  conclusion.  Because  the  collected 
data  is  a  random  sample  of  a  larger  population,  it  is  very 


likely  there  will  be  some  discrepancy  between  the  value  of  a 
sample  parameter  and  its  "true"  value. 

Generally  a  is  chosen  to  be  .01  or  .05  which  results  in 
a  99X  or  95X  confidence  level,  respectively.  It  must  be 
noted  that  as  confidence  goes  up  the  probability  of  accepting 
HQ  goes  up,  regardless  of  whether  Hq  is  true  or  not.  The 
probability  of  accepting  Hq  when  it  is  indeed  false,  a  type 
II  error,  is  denoted  by  8.  For  a  given  sample  of  data,  if 
the  probability  of  rejecting  Hq  when  it  is  true  (a)  gees  down, 
the  probability  of 'accepting  Hq  when  it  is  false  (6)  goes  up. 
Both  a  and  g  are  affected  by  the  amount  of  data  collected; 
as  sample  size  goes  up,  8  will  drop  for  a  given  a,  or  a  can 
be  lowered  while  8  is  unaffected.  Since  sample  size  is  con¬ 
strained  due  to  costs  in  OT&E,  reasonable  values  of  8  are  not 
possible  at  low  significance  levels  (a) .  When  dealing  with 
"small"  samples,  it  is  best  to  set  a  as  high  as  is  comfort¬ 
able,  without  loss  of  credibility.  In  operational  tests,  a 
as  high  as  .2  can  be  used.  For  further  discussion  of  signi¬ 
ficance  and  power  see  (Ref  7:389-390),  (Ref  3:284-288). 

When  using  statistical  methods  to  analyze  OT&E  data, 
there  are  three  distinct  areas  of  interest: 

1)  estimation  of  a  MOE,  e.g.  CEP,  Probability  of  Kill 

(PK> ; 

2)  determination  of  a  confidence  interval,  i.e.  finding 
the  interval  about  an  estimate  which  has  a  known  probability 
of  including  the  "true"  mean  value  of  the  MOE; 
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!  3)  hypothesis  testing,  i.e.  comparing  sample  values  of 

j  two  or  more  MOE's  or  comparing  one  value  against  a  known  or 

assumed  value. 

i 

Hypothesis  testing  is  the  main  statistical  tool  used  in 
l 

OT&E  data  analysis.  The  hypotheses  under  consideration  can 
take  on  many  different  forms.  Two  of  the  most  common  are: 

1)  hypotheses  concerning  population  parameters,  e.g. 

a)  HQ:  CEP  >.  100  feet 

H. ;  CEP  <  100  feet 
A 

b)  H  :  MTBF  (component  A)  =  MTBF  (component  B) 

o 

H  :  MTBF  (component  A)  ^  MTBF  (component  B) 

A 

2)  hypotheses  concerning  the  relationships  among  a 
response  variable  (MOE)  and  one  or  more  independent  variables, 
e.g. 

Given;  y  =  Bc  +  BjX^  +  B2X2  +  e 

y  ;  detection  range 

X^:  target  altitude 

X2:  target  airspeed 

B  ,  B. ,B0:  coefficients 
o  12 

e  :  random  error  term 
Ho:  b1=b2=0 

Detection  range  is  not  a  linear  function  of  target 
altitude  or  airspeed. 

Ha;  B^O  or  B^O  or  B^O  and  B2^0 
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Detection  range  is  a  linear  function  of  either  tar¬ 
get  altitude,  target  airspeed  or  both. 

The  estimation  of  a  population  parameter  from  a  sample 
of  data  can  be  carried  out  by  common  mathematical  equations. 

In  this  guide  these  parameters  include: 

1)  sample  mean  (x) ,  an  estimate  of  the  population  mean 

(v); 

o 

2)  sample  variance  (s  ) ,  an  estimate  of  population 

2 

variance  (a  ) ; 

3)  and  sample  probability  of  success  (p) ,  an  estimate 
of  the  population  probability  of  success  (p) . 

The  term  confidence  interval  applies  to  the  range  of 
values  about  a  sample  estimate  which  has  a  known  probability 
(1-a)  of  including  the  true  population  value.  Because  the 
data  sample  is  a  random  selection,  a  confidence  interval  can 
be  calculated  which  takes  into  account  the  randomness.  The 
larger  the  confidence  level  (1-cOxlOOX,  the  wider  the  interval 
will  be. 

To  determine  the  necessary  sample  size  for  a  given  con¬ 
fidence  level,  the  "maximum  error  of  the  estimate".  E  ,  must 
also  be  known.  The  maximum  error  (E)  is  the  largest  acceptable 
difference  between  the  estimate  and  the  population  value 
which  is  measurable.  For  a  given  significance  level  (a) ,the 
sample  size  (n)  must  be  increased  if  the  maximum  error  is 
decreased. 


ical  Terms 
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MOE: 

Ho! 

ha! 

a: 


measure  of  effectiveness 

stated  or  null  hypothesis 

alternate  hypothesis 

significance  level  -  probability  of  a 
type  I  error 


Confidence  level:  (l-a)xlOOX 

Type  I  error:  reject  Hq  when  it  is  true 


3: 

Power : 


probability  of  type  II  error 
•1-3 


Type  II  error:  fail  to  reject  Hq  when  it  is  false 


n: 

E: 

x 


i* 


y : 
2 


a  : 

Pi 


sample  size 

maximum  error  of  the  estimate 

ith  sample  observation 

sample  mean,  an  estimate  of  p,x  =  ^ 


i  _ 


population  mean 

sample  variance,  an  estimate  of 

2  2  n  2  —2 
o  ,  s  =  (  I  xf  -  nx  )/(n-l) 

i=l  1 

population  variance 

sample  probability  of  success,  an  estimate 
of  p,  p  =  (ft  of  successes)/!) 

population  probability  of  success 


Relationship 

-  For  constant  n,  an  increase  in  a  decreases  3,  and  vice  versa 


For  constant  3,  an  increase  in  n  decreases  a 


n  m3 


V.  MEASURES  OF  EFFECTIVENESS 

Recent  OT&E  reports  have  made  use  of  three  MOE  types: 

1)  Proportions 

2)  Averages 

3)  Measures  of  Dispersion 

These  types  of  MOEs  will  adequately  measure  most  objectives 
to  be  tested;  however,  in  no  way  do  they  comprise  all  of  the 
possibilities.  Analysis  of  only  these  MOE  types  will  be 
covered  in  this  guide. 

Proportions 

Examples  of  measures  which  are  proportions  are: 

.  1)  Probability  of  kill  <Pk) 

2)  Percent  of  total  launches  that  are  successful 

3)  Attrition  rate 

4)  Number  of  targets  sighted  vs.  number  present 
Proportions  can  be  binomial  or  mult inoinial  measures  depending 
on  the  number  of  possible  outcomes  of  a  single  trial.  The 
binomial  trial  has  only  two  outcomes  possible,  e.g.  success 
or  failure  and  kill  or  no  kill,  while  the  multinomial  trial 
can  have  more  than  two  outcomes ,  e.g.  win,  lose,  draw  and  kill, 
no  kills,  killed.  The  MOE  associated  with  binomial  trials 
will  generally  be  the  probability  associated  with  one  of  the 
outcomes,  whereas  in  the  multinomial  case  the  MOE  could  be 

the  probability  of  one  or  more  outcomes  as  well  as  the  ratio 
between  outcomes  or  groups  of  outcomes.  The  sum  of  all 
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probabil;'  .ies  of  all  outcomes  must  add  up  to  one  in  both  the 
binomial  and  multinomial  experiments.  The  assumptions  which 
most  affect  tests  of  proportions  are  the  assumptions  that 
the  probability  of  an  outcome  remains  constant  and  that. the 
trials  are  independent,  which  means  that  the  outcome  of  a 
trial  is  not  affected  by  the  outcomes  of  previous  trials. 

Both  distributions  are  fully  specified  by  the  total  number 
of  observations,  n,  and  p^,  the  probability  of  the  ith  out¬ 
come,  for  all  i  . 

Tabled  prooabilities  are  available  for  the  binomial  dis¬ 
tribution  for  n  up  to  20  and  for  p  a  multiple  of  .05.  When 
testing  below  n  of  20,  a  linear  interpolation  of  the  tabled 
values  is  acceptable  (Ref  15:183),  but  the  best  approach  with 
small  n  is  to  compute  the  probability  exactly.  When  n  is 
greater  than  20,  the  normal  approximation  is  acceptable  for 
p  between  .1  and  .9  and  the  Poisson  approximation  can  be  used 
when  p  or  1-p  is  less  than  .1. 

Binomial 

two  outcomes  (success/failure) 

p:  the  probability  of  success 

q=l-p:  the  probability  of  failure 

n:  total  number  of  observations 

y:  total  number  of  success 

6  =s  estimate  of  p 

n 
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Exact:  P(y=k)  =  (£)pk  qn_k  (probability  that  y=k) 


P(y<k)  =  Z  (?)  p1  qn_i 
i=0  1 


/tu  „  nl 
(k}  "(n-k)  l  kl 


Normal  Approximation:  Z  **  k  ~  np 

*/npq 

If  2  >  0(  P(y<k)  =  F  (Z)  (Ref  15:127) 

If  2  <  0,  P  (y  <k)  *  l-F(-Z)  (Ref  15:127) 

The  ±  H  is  a  continuity  correction  to  be  used  if  n<100, 
Use  -  when  k<np  and  +  \  otherwise. 

Poisson  Approximation:  \  =  np,  x*  =  k 

P(y>k)  *  T (x* , X)  (Ref  15:213) 

P(y£k)  =  1-T (x’  +1 ,  X)  (Ref  15:213) 

Multinomial 

k  possible  outcomes 

n:  total  number  of  observations 

p^:  the  probability  of  the  ith  outcome 

* 
b 

* 

i=l 

y^  =  total  number  of  observations  of  outcome  i 
*i  =  yi 
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Averages 


Some  examples  of  MOE's  which  are  averages .  or  means . 

are: 

1)  average  detection  range 

2)  mean  miss  distance  (MMD) 

3)  mean  time  between  failures  (MTSF ) . 

These  measures  are  the  most  common  to  OT&E,  and  the  statis¬ 
tical  tests  are  generally  known.  Two  basic  approaches  are 
covered  in  dealing  with  averages.  The  distribution  of  the 
measure  can  be  assumed  1)  to  be  normal  -  that  is,  continuous, 
unimodal  and  bell- shaped  symmetrically  about  the  mean  or  2) 
to  be  non-normal.  Nonparametric  tests  must  be  used  if  the 
population  is  assumed  to  be  non-normal  or  distribution-free. 
The  proper  parametric  or  nonparametric  test  is  determined  by 
the  number  of  samples  to  be  tested.  Linear  regression  and 
analysis  of  variance  (ANOVA)  can  be  used,  with  data  assumed 
to  be  normal,  to  determine  the  relationship  of  the  MOE  to 
other  factors  or  independent  variables. 

Normal  Distribution 

unimodal:  one  "hump”  or  "peak" 

continuous:  data  point  y ^  can  take  on  all  values 

symmetric:  the  probability  function  is  a  mirror  image 

about  the  mean  (y) 

interval:  -“  to  °°,  the  true  normal  distribution  cannot 

be  absolutely  restricted  from  taking  on  any 
value  although  the  probability  of  taking  on 
values  in  extreme  regions  is  essentially  zero. 
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X  >  0 


p(y<  <  x)  =  f  (x) 


(Ref  15:127) 


x  <  0 


P(y,  <  x)  =  l-F(-x)  (Ref  15:127) 


Measures  of  Dispersion 

The  primary  measures  of  dispersion  are  circular  error 
probable  (CEP) ,  range  error  probable  (REP)  and  deflection 
error  probable  (DEP) .  These  three  measures  are  all  based  on 
the  assumption  that  the  data  points  are  distributed  normally 
on  one  (REP, DEP)  or  both  axes  (CEP) .  If  this  is  not  found 
to  be  the  case,  the  mean  miss  distance  (MMD)  can  be  used  as 
another  measure.  The  CEP  measure  assumes  that  the  dispersion 
along  both  the  range  and  deflection  axes  is  normally  distri¬ 
buted  with  equal  variances,  that  is,  the  pattern  is  circular 
normal.  The  REP  and  DEP  measures  were  developed  to  account 
for  dispersion  patterns  which  are  elliptical  in  shape.  Cir¬ 
cular  error  probable  can  be  thought  of  as  the  radius  of  the 
circle  which  encompasses  half  of  the  population's  points. 
Because  CEP  is  the  most  accepted  measure  of  dispersion  for 
bombing  or  navigation  accuracy,  it  will  often  be  required 
even  if  the  pattern  is  not  circular.  When  computing  CEP, 

REP,  or  DEP,  the  mean  impact  point  (MPI)  should  be  used  and 
this  position  should  be  reported  with  respect  to  the  target 
center  or  desired  mean  impact  point.  REP  and  DEP  are  similar 
to  CEP  in  that  they  give  borders  in  which  50*  of  the  impacts 
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should  occur  on  one  axis. 


x^s  lateral  miss  distance 
y^:  longitudinal  miss  distance 
MPI:  X,  y;  mean  point  of  impact 


REP 

DEP 


.6745 


For  non-circular  patterns  a  50%  circle  can  be  computed  using 
these  formulas: 

k  ‘  sx/sy  ' 

If  k  >.  .3,  CEP  a  .6152  s  +  .5640  s 

x  y 

If  k  <  .  3/  CEP  =  (.82k  +  .007)  s  +  .6745  s 
~  k  y 

*NOTE:  These  formula  assume  s  >s  if  not,  reverse 

y  x 

s  /S  in  all  formulas, 
x  y 

When  dealing  with  circular  patterns  the  50%  circle  can  be 
computed  as: 

CEP  -  1.1774 


A  more  generalized  equation,  making  use  of  the  Rayleigh  dis¬ 


tribution,  for  the  probability  distribution  of  the  radial 

/*2  2 

miss  distance,  /X  +Y  ,  is: 


P(r  <.  R)  =  1 


(!4)-R2/CEp2 


r:  radial  miss  distance 
R:  a  specific  radius 


For  a  complete  discussion  of  CEP,  REP,  and  DEP  see  (Ref  28) . 
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VI.  DATA  ANALYSIS 


The  procedure  which  OT&E  analysts  have  used  to  proceed 
from  the  planning  phase  to  writing  the  final  report  is  not 
nething  that  is  easily  standardized.  However,  every  analy¬ 
sis  has  certain  aspects  which  are  always  present.  The  objec¬ 
tive  is  to  provide  a  procedure  which  is  general  enough  to 
cover  all  aspects  of  OT&E  analysis  and  yet  be  specific  enough 
to  be  used  as  a  ready  reference  by  all  analysts. 

The  basic  steps  in  an  analysis  are: 

A.  SPECIFICATION 

1)  Determine  the  objective  of  the  analysis. 

2)  Determine  the  MOE  that  will  best  quantify  the 
results  of  the  analysis  and  best  relates  to  the 
objective. 

B.  EXPERIMENTAL  DESIGN 

1)  State  the  hypothesis  (H0)  for  which  rejection  is 
desired  and  the  alternative  (HA)  which  satisfies 
the  objective. 

2)  Determine  what  level  of  confidence  (  (l-a)xlOO%) 
is  required. 

3)  Determine  which  statistical  test  is  most  suited 
to  the  MOE  and  stated  hypothesis  (H0) . 

4)  Determine  the  sample  size  required  based  on  the 
significance  level  (a)  and  the  maximum  error  of 
the  estimate  (E) .  (Mot  always  possible) 

C.  DATA  COLLECTION 

1)  Collect  the  data,  following  the  experimental 
design  closely,  and  keeping  the  assumptions  of 
the  selected  test  in  mind. 
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D.  ANALYSIS 


1)  If  possible,  test  each  of  the  assumptions  of 
the  chosen  test  with  the  collected  data. 

2)  Using  the  results  of  the  test  of  assumptions, 
determine  which  test  is  best  suited  for  the  data. 

3)  Apply  the  statistical  test  at  the  confidence 
level  originally  stated. 

4)  Interpret  the  results  of  the  test,  stated  in 
terms  of  rejecting  or  failing  to  reject  the 
stated  hypothesis  <H0) . 

E.  REPORTING  RESULTS  AND  CONCLUSIONS 

A.  Specification 

Determination  of  the  objective  and  the  MOE  are  completed 
with  the  publication  of  the  Test  Plan.  Three  criterion  should 
be  applied  in  the  decision: 

1)  operational  relevance 

2)  measurability 

3)  quantif iability . 

B.  Experimental  Design 

The  design  of  the  experiment  is  the  most  critical  phase 
of  the  analysis  in  terms  of  getting  the  most  out  of  the  re¬ 
sources  expended.  The  number  of  replications,  or  sorties,  of 
each  kind  should  be  based  on  analytical,  not  operational  con¬ 
siderations.  This  cannot  be  done  without  a  complete  design, 
accomplished  before  the  data  is  collected.  Once  the  MOE, 
hypothesis  and  confidence  level  are  determined  the  appropriate 
statistical  test  must  be  selected. 


1 .  Proportions 

The  analysis  of  MOE's  which  are  proportions  can  be 
accomplished  in  three  basic  ways.  If  the  experiment  is  in¬ 
deed  binomial  with  two  outcomes  and  a  constant  probability 
of  success  (p),  the  exact  binomial  or  an  approximation  can  be 
used.  If  the  probability  of  success  is  changing,  linear  re¬ 
gression  can  be  used  to  determine  the  behavior  of  p  as  a 
function  of  other  variables.  In  the  case  of  multinomial 
experiment  (more  than  two  outcomes),  the  Chi-square  contin¬ 
gency  table  can  be  used  to  determine  equality  of  p^  across 
samples. 

2.  Averages 

The  analysis  of  MOEs  which  are  averages  can  be 
carried  out  using  either  parametric  or  nonparametric  (dis¬ 
tribution-free)  statistical  tests.  Both  methods  have  certain 
advantages  and  disadvantages;  however,  the  choice  should  be 
based  solely  on  the  actual  or  expected  characteristics  of  the 
data.  For  a  complete  discussion  of  the  subject  see  (Ref  9: 
30-34} ,  (Ref  10:91-93) . 

The  disadvantages  of  nonparametric  statistical  tests 
include: 

a)  When  the  assumptions  of  the  parametric  tests  are 
met,  for  a  given  sample  size  and  significance  level 
nonparametric  tests  are  often  not  as  powerful  as 
the  best  parametric  test,  that  is,  the  probability 
of  failing  to  reject  an  erroneous  HQ  is  greater. 

b)  Required  sample  sizes  cannot  be  estimated  accurately 
for  any  nonparametric  test. 
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c)  Nonparametric  tests  are  not  as  sensitive  to  small 
deviations  in  some  distribution  measures,  e.g.  shape, 
position. 

d)  Nonparametric  tests  are  less  common,  there  are  fewer 
references,  and  there  are  fewer  people  well  acquainted 
with  their  use. 

The  advantages  of  nonparametric  statistical  tests  in¬ 
clude: 

a)  Nonparametric  tests  have  fewer  and  less  restrictive 
assumptions  which  make  them  applicable  to  a  wide 
variety  of  data. 

b)  Nonparametric  tests  can  be  used  to  compare  measures 
from  differing  distributional  types. 

c)  Nonparametric  tests  can  be  used  with  ranked  (ordinal) 
and  categorical  (nominal)  data. 

d)  Nonparametric  tests  can  be  used  when  nothing  is 
known  about  the  underlying  distribution. 

e)  Nonparametric  statistics  often  have  relatively  sim¬ 
ple  mathematical  derivations  and  the  development  is 
more  easily  understood. 

f)  Nonparametric  statistical  tests  must  be  used  unless  the 
population  distribution  is  known  to  be  normal  (Ref  9:32). 

While  it  might  seem  that  nonparametric  statistics  have 
more  general  applicability,  the  parametric  tests  presented 
here  will  generate  accurate  results  even  with  minor  deviations 
from  some  assumptions.  The  choice  of  a  particular  test  within 
each  category,  parametric  or  nonparametric,  is  very  similar 
in  that  the  choice  is  usually  based  on  the  number  of  samples. 

The  parametric  tests  used  here  will  all  assume  a  normal  dis¬ 
tribution. 


The  one  and  two  sample  tests  of  means  are  based  on 


comparing  sample  moans  and  determining  the  probability  that 
they  are  statistically  different.  The  analysis  of  variance 
(ANOVA)  and  regression  methods  ’re  based  on  the  linear  rela¬ 
tionship  between  a  dependent  variable  and  other  independent 
variables. 

3.  Measures  of  Dispersion 

Use  of  CEP,  REP,  DEP  and  variance  measures  are 
generally  confined  to  the  analysis  of  weapons  delivery  or 
navigation  accuracy.  The  analysis  of  means  can  be  applied 
to  accuracy  scores',*  however,  the  standard  practice  is  the 
use  of  CEP.  The  CEP  is  calculated  about  the  midpoint  of  the 
dispersion  pattern  (MPI) .  The  relative  position  of  the  MPI 
to  the  target  can  be  considered  system  bias. 

With  no  prior  knowledge  of  system  accuracy  or  dis¬ 
persion  patterns  it  will  be  best  to  assume  a  non-circular 
pattern  and  use  the  adjusted  CEP  formula  for  elliptical  pat¬ 
terns.  While  the  delivery  of  free-fall  ordnance  from  jet 
aircraft  will  almost  always  result  in  elongated  patterns, 
navigation  accuracy  and  guided  weapons  may  in  fact  have 
circular  distributions. 

The  difference  between  the  MPI  and  the  target  will 
only  be  true  system  bias  when  all  other  errors,  (e.g.  aiming 
error)  have  been  accounted  for  and  corrected. 

4 .  Determining  Sample  Size  Required  (Nr) 

In  many  cases  it  is  not  possible  to  determine  how 
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many  samples  will  be  required  for  a  given  combination  of  test, 
significance  level  (a)  and  maximum  error  of  the  estimate  (E) .  . 
When  planning  on  the  use  of  nonparametric  statistics  there 
are  no  sample  size  calculations  available.  However,  for 
some  parametric  tests  the  required  sample  size  (Nr)  is  easily 
calculated. 

TEST  Nr 

"™  2 

One  sample  test  of  proportion  N  =  £(l-£)  [W  /E] 

IT  d 

W  =x:  F(x)=l-a  (Ref  15:127) 
aa=  ax/2 

E:  maximum  error  of  the 
estimate 

„  £:  an  estimate  of  p  (If  no 

estimate  is  available  use 

One  sample  test  of  a  mean  N  =  (s  •  W_,  /E) z 

r  r ,  n 

W_  *  t  (Ref  15:283) 

*  i  n 

F  =  i-K\  :  I 
n  =  a 

A 

s  =  an  estimate  of  a 
2 

Note:  If  a  is  known  substi¬ 
tute  a  for  s 

Two  sample  test  of  means  Nr^  =  r 

W_  =  t  (Ref  15:283) 

r  ,  n 

F  =  1-a^ 
n  =  a 

2 

Note:  If  is  known  substi¬ 

tute  a ^  for  s  ^ 
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C .  Data  Collection 


The  process  of  collecting  data  for  OT&E  is  carried  out 
in  many  ways  and  is  not  always  an  analyst's  responsibility. 

The  analyst  should  be  given  some  input  because  the  quality 
of  any  analysis  is  only  as  good  as  the  data  upon  which  it  is 
based.  For  this  reason  the  analyst  should  do  all  that  can  be 
done  to  see  that  data  collection  is  carried  out  properly. 

This  can  be  accomplished  by  participating  in  the  collection 
and  by  verifying  the  quality  of  all  data  by  inspection. 

D.  Analysis 

The  data  analysis  can  be  divided  into  three  basic  tasks: 

1)  testing  the  assumptions 

2)  testing  the  stated  hypothesis  (Hn) 

3)  interpreting  the  results  . 

1)  Testing  the  assumptions 

The  test  of  assumptions  are  those  tests  which  deter¬ 
mine  whether  the  test  selected  during  the  experimental  design 
phase  is  in  fact  the  proper  test.  Unlike  the  testing  of 
hypotheses  in  the  next  section,  the  hypothesis  when  testing 
assumptions  will  be  that  the  assumption  is  valid.  Therefore 
assumption  tests  will  not  detect  some  deviations  from  the 
assumptions . 
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Assumption 

Test 

References 

Randomness 

Runs  Test  (pp  A-3 5) 

(9:52-58) 

(14:34-38) 

Proportions 

«• 

Constant  p 

Chi-Square  goodness- 
of-fit  Test  (pp  A-52) 

(9:42-47) 

(10:190-198) 

Averaaes 

Normality 

Lilliefors  Test 
(pp  A-57) 

(10:357) 

Chi-Square  goodness- 
..  of-fit  Test  (pp  A-52) 

(9:42-47) 

(10:190-198) 

Known  variance 

Chi-Square  Test 
(pp  A-43) 

(5:75) 

(7:396) 

Equality  of  two 
variances 

F  Test  (pp  A-43) 

(5:90) 

(7:400) 

Equality  of  k 
variances 

Levene's  Test 

Bartlett's  Test 

(5:253) 

(5:252) 

(7:510) 

Measures  of  Dis- 
persion 

Normality 

Lilliefors  Test 
(PP  A-57) 

(10:357) 

MPI  =  Target 

F  Test 
(pp  A-43) 

(7:400) 

(27:2) 

X-Y  Correlation 

t  Test 

(27:2) 

o\  Hvriothesis  Testinq 

Tests  of  hypotheses  can  be  formulated  in  many  ways; 


in  this  guide  the  most  general  cases  will  be  used  to  ensure 
broad  applicability. 
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a)  One  sample  case 

(1)  Hc:  |0-Q* |<E 

Ha:  | Q-Q* | >E 

(2)  Hq:  Q<Q*+E 

Ha:  Q>Q*+E 

(3)  H  :  Q+E>Q* 

o  — 

Ha:  Q+E<Q* 

These  hypotheses  allow  comparison  of  a  population  measure 
or  parameter,  Q,  against  a  hypothesized  value,  Q*,  with  a 
meaningful  difference,  E.  The  difference,  E>0,  is  the  max 
error  of  the  estimate  when  the  objective  is  to  detect  any 
significant  difference.  Case  (1)  is  called  a  two-tailed  test 
because  differences  are  detected  when  the  sample  measure  is 
significantly  different,  larger  or  smaller,  than  the  hypoth¬ 
esized  value,  Q* . 

b)  Two  sample  case 

(4)  Hc:  \QX  -  Q2|<E 

HA:  |Q1  “  Q2I>E 

(5)  H  :  Q.  <  Q,+E 

O  1  i 

HA:  Q1  >  Q2+E 

Note:  estimate  of  >.  estimate  of  Q ^ 

These  hypotheses  allow  the  comparison  of  two  populations ' 
measures,  and  Q2.  By  setting  E=0  the  hypotheses  reduce  to 
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a  direct  comparison  of  the  measures:  (4)  H  :  Q  =Q„,  (5)  H  : 

0  12  O 

Ql-°2‘ 

c)  More  than  two  sample  case 

(6)  Ho:  Q1=Q2*...Qk  or 

H.:  At  least  one  of  the  measures  is  not 
equal  to  all  others 

(7)  Ho:  Bl=B2=...Bk=0 

Ha:  At  least  one  of  the  slope  coefficients, 
B^,  is  not  equal  to  zero. 

In  hypothesis  (6)  the  comparison  is  made  among  all  sample 
measures  for  equality.  Hypothesis  (7)  pertains  only  to  tests 
where  linear  regress-ion  is  used. 

d)  Hypotheses  for  testing  assumptions 

(8)  Hq:  The  assumption  holds 

H  :  The  assumption  does  not  hold. 

A  1 

Hypotheses  for  tests  of  assumptions  are  constructed  so 
that  the  burden  of  proof  is  on  rejecting  the  assumption.  Un¬ 
like  other  testing,  the  hypothesis  is  not  made  so  that  the 
analyst  wants  to  reject  Hq. 

The  acceptance  region  is  the  range  of  values  of  the 
test  statistic,  T,  which  do  not  support  rejection  of  the  Hq. 
The  acceptance  region  differs  with  the  type  of  hypothesis 


and  the  particular  test  employed. 
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Reporting  Results 

The  final  step  in  this  guide  is  the  reporting  of  conclu-  . 
sions  based  on  the  results  of  the  statistical  test.  The  in¬ 
terpretation  of  the  results  must  be  left  to  the  tester ;• how¬ 
ever,  the  conclusions  reached  from  the  tests  can  be  generalized. 
It  should  be  remembered  that  no  statistical  test  has  ever 
proved  anything;  they  do  lead  the  tester  to  conclusions  con¬ 
cerning  the  probability  that  the  stated  hypothesis  (H0)  is 
or  is  not  true. 
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VII.  STEP- BY-STEP  GUIDE 


The  following  step-by-step  guide  will  cover  the  analysis 
of  test  data  from  the  initial  tests  of  assumptions  to  the 
reporting  of  the  conclusions.  Much  of  the  procedure  is 
carried  out  in  a  flow  chart  which  may  be  confusing  at  first 
use.  To  overcome  any  potential  problems  it  is  best  to  work 
through  the  overall  example  (ppA-82)  before  using  the  guide 
to  analyze  a  real  problem. 

Step  I: 

Graph  the  collected  data  to  detect  any  obvious  trends. 
Step  II: 

Select  a, ,  the  significance  level  for  the  test  of  hypoth¬ 
esis  for1 the  MOE. 

Select  a_,  the  significance  level  for  the  test  of  assump¬ 
tions. 

Step  III: 

Test  each  sample  for  randomness 
Hq:  The  sample  is  a  random  sample 
Hq:  The  sample  is  not  a  random  sample 

RUNS  TEST 

Proportion  measures:  assign  "+"s  to  one,  or  a  group  of, 
outcome (s);  assign  "-"s  to  all  other  outcomes. 

Other  measures:  assign  a  to  each  observation  greater 
than  the  mean,  assign  a  to  all  other  observations, 

n^  =  no.  of  +'s  n^  =  no.  of  - ' s 
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run-,  "a  succession  of  identical  symbols  which  are  fol¬ 
lowed  and  preceded  by  different  symbols  or  no  symbol  at  all" 
(Ref  9:52) 

Arrange  the  +  *s  and  -'s  in  the  order  that  the  correspond¬ 
ing  data  was  collected.  Count  the  number  of  runs. 

T  =  the  number  of  runs 

L  =  ^^a  (Ref  15:415) 

U  =  U  . 

n^n^  1-a 

a  =  a2/2 

Acceptance  region:  L<T<U 

If  the  Hq  is  rejected  the  data  should  be  discarded  and 
new  data  should  be  collected.  If  this  is  not  done  the  sta¬ 
tistical  tests  are  no  longer  valid  and  may  lead  to  erroneous 
conclusions . 


Binomial  Confidence  Interval 


P(PL  <  P  <  Pu)  -  l-a 
Pl  k+(n-k+l)P1_ a/2,2(n-k+l) ,2k 

p  =  (k+1>Fl-«/2,2(k4-l)  ,2(n-k) 

(n-k)  +  (k+1)  "  2  (k4-i ) ,  2  (n-k) 

P 

a,n^,n_  is  the  tabled  F  value  from  (15:305) 
Ref:  (17) 
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non 


PARAMETRIC  TESTS  Continued 


Paired  SimtUi 


iHypochaaea  (x, >x,) 

iK-W2l<E  1_2 


H-i  a-aj/2 
M^j+S 

H-2  ft. 


V»cha  atandard  dev  la  cion  of  Che 
difference  of  the  palra 

W-t  , 
a,a-l 

at  eha  nuaber  of  palra 
Rafaraacaa:  (3:338)  (5:84) 


1il*u-*2i /o 


Iod€P#pd«nc  S— pl«> 


°l  A 

V  -  _L  ♦ _ £ 

fll  “2 

U-i:  F(x)*l-a 

(13:127) 

I  Rafaraacaa: 
(3:328) 


Accepcance  Region 


?<T7V+E 


_  2  2 

**1  *2 

Teat  equal icy 
of  varlancea 

V  «H 

CO  TO  pp  A-43 


CO  TO  8 
pp  A-80 


•^2)—araj 


acc  H0j) 


(15:283); 


Rafaraacaa: 

(3:337)  (7:390) 
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1  .  ^  #a  'a  V-.  *  f  a  *  «  *  A  ^a  *  I 


afl  *i  2  faa 


-  iA  la  -  •-  •— A  -  L  i_ 


Confidence  Intervals 


Cl  about  a  point  estimate 

P  (X-W*v  <.  y  <.  X  +  w*V)  =  1-a 

If  a2  is  known:  W*  =  X:  F(X)  =  l-a/2 
2 

If  a  is  unknown:  W*  =  ta/2  n-1 


Cl  about  a  difference 

P(x1  -  X 2  -  W*V  <  ^  -  y2  <  ^  -  X2  +  W*V)  =  1-a 
Paired  Data 

Di  =  (Xli“X2i)?  1  ~  l,2,...n 
If  a2  -is  known:  W*  =  X:  F(X)  =  l-a/2 
2 

If  is  unknown:  W*  *  fca/2  n  1 


Independent  Samples 


If 

°|  _y 
*1  *2 

is 

known:  W*  = 

X:  F  (X) 

=  l-a/2 

If 

_2 

°x  —X 
X1  *2 

is 

unknown:  W* 

=  V/2, 

nl+n2"2 
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HOKPaRAMETRIC  tests 


JL 


ioth«»« 


AalipndttcV 

\S»npl«»7  / 


NO 


„s 

CO  TO 

pp  A-67 

KROSKAL- 

I  WALLIS 

CO  TO 
PP  A-72 
FRIEDMAN 


MEASURES  OF  DISPERSION  Cooeinutd 


Ur.oer  Confidence  Limit  for  CEP 

2 

P  (CEP  <.  (estimated  CEP)  (1+k  )  (n-l)_  ) 


W 


a 


w  -  *2af,a 

a:  the  desired  confidence  level 

df  =  INT  ( (1+k2)  (n-1) 3 
Reference  (28:6) 
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Contingency  Table 


The  contingency  table  is  used  to  test  hypotheses  con¬ 
cerning  the  probability  of  the  occurrence  of  particular  out¬ 
comes  between  populations.  This  is  of  special  use  in  the 
multinomial  experiment  where  the  hypotheses  is  that  the  pro¬ 
bability  of  outcome  j  is  equal  among  all  populations  for  all 


outcomes .  The 

"c"  outcomes. 

test  is  generalized  for 

For  further  discussion 

outcome  1  outcome  2  . . 

"r"  populations  and 

see  (Ref  10:153-158) 

.  outcome  c  total 

population 

1 

°n 

°12 

o 

o 

nl 

population 

» 

• 

2 

\ 

H 

CM 

O 

°22 

o 

to 

o 

n2 

• 

population 

r 

°rl  . . 

°r2 

°rc 

n„ 

r 

total 

*1 

rt 

N) 

tc 

N 

°ij  =  the  numt5er  times  outcome  j  is  observed  in  the  ith 
population. 

tj  =  the  column  total  of  0^ j ' s  for  all  i 
n^  =  the  row  total  of  0^^ ,  for  all  j 

If  A-l:  C=  the  number  of  possible  outcomes  or  groups  of 
outcomes,  r=  the  number  of  samples  to  be  compared. 

If  A=2:  C=2  (Binomial) ,  r  =  the  number  of  samples  to  be 
compared. 
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Hypotheses 

V  Plj  e  P2j  *  prj  for  a11  J 

The  probability  of  outcome  j  is  equal  across  all  popula¬ 
tions  for  all  outcomes. 

Hx:  At  least  two  of  the  populations  (P,  P,.  ...  P  . )  are 

a  23  rj 

not  equal  within  an  outcome. 


Test 

The  test  statistic,  T,  is  calculated  as: 


T 


z  2  (0ii)2N 

i=l  J=1  c^. 


N 


W  =  (Ref  15:294) 

f  ,n 

r  -  l-ax 

n  «  (r-1) (c-1) 

Acceptance  Region 
T  <  W 

GO  TO  Reporting  Results  -  pp.  A-78 
Example: 

Determine  whether  or  not  the  probability  of  kill  (P  ) 

J\ 

is  affected  by  missile  type  (AIM-9J,  AIM-9L,  AIM-9M)  at 
a^  =  .05. 

Hypothesis : 

Ho:  Pk(9J)  =  pk(9L)  *  pk<9M> 

H„ :  At  least  two  of  the  missiles  have  different 
A  ^  .  _ 
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Data : 


Outcome  of  Trial 


Missile  Type 
9J 
9L 
9M 

Total 


Kill 

10 

13 

14. 

37 


No -Kill 
8 
4 
_4 
16 


Total 

18 

17 

li 

53 


Test: 

T  «  102*53  .  132 • 53  .  142*53  .  82*53  .  42*53  .  42*53 
37*18  +  37 • 1 7  37*18  1 6 • 1 8  +  16*17  16*18 

T  =  2.635 

W  =  X2  n  (Ref  15:294) 

F  =  1-.05  =  .95 
n  *  (3-1)  (2-1)  =  2 
W  =  5.991 


-  53 


T  <.  W  therefore  we  fail  to  reject  Hq  at  a^=.05. 

There  is  no  statistical  evidence  to  support  a  claim 
that  missile  type  affects  at  a^=.05. 
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I  i 

U1 


The  Chi-Square  goodness-of-f it  test  is  used  to  determine 
if  a  binomial  experiment  has  a  constant  probability  of  suc¬ 
cess  across  time.  This  test  is  similar  to  the  contingency 
table  however  there  is  only  one  row.  Each  cell  should  con¬ 
tain  five  samples  or  more.  The  data  must  be  ordered  by  time 
and  divided  into  cells.  The  observed  frequency  (0^)  is  equal 
to  the  number  of  successes  in  cell  i .  The  expected  frequency 
(Ei)  is  equal  to  £>,  the  total  successes  divided  by  total 
observations,  time's  the  number  in  cell  i. 
n^  =  number  in  cell  i 
0±  «=  successes  observed  in  cell  i 

E  =  k 
1  n  *  ni 

k  -  total  observed  successes 
n  =  total  sample  size 
m  =  no.  of  cells 


Hypothesis 

Hq:  p  is  constant  across  time,  i.e.  PiscP2=#  •  *Pra 
Ha:  p  is  not  constant  across  time 


Test 


m  2 

I  (O.-E.r/E. 
i*l  1  1  1 


W 

F 


F,  n 
l-a„ 


(Ref  15:294) 


n  *=  m-1 
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Linear  regression  is  very  similar  to  ANOVA,  indeed  ANOVA  . 


is  a  special  case  of  regression.  Linear  regression  attempts 

to  determine  the  linear  relationship  between  the  dependent 

variable  (MOE)  and  any  number  of  independent  variables  which 

affect  its  value.  The  technique  estimates  the  coefficients 

(B^)  of  the  linear  equation  (y=B0  +  B^Xi  +  B2X2)  which  best 

fits  the  data.  The  "best"  line  is  defined  as  the  one  for 

which  the  sum  of  the  distances  squared  (SSE)  from  the  observed 

values  of  the  dependent  variable  (y)  to  the  estimated  line 

(y)  is  minimized.  The  measure  of  worth  of  the  model  is  R^ 

which  is  a  function  of  how  much  the  SSE  for  the  estimated 

* 

line  (£)  differs  from  the  SSE  about  the  line  (y) .  An 

close  to  zero  indicates  little  additional  explanation  of  the 

dependent  variable  by  the  independent  variables.  It  should 

2 

be  noted  that  R  will  always  increase  when  variables  are 

added  and  that  a  better  measure  of  explanatory  power  is  the 
2  — 2 

adjusted  R  (R  )  which  is  adjusted  for  the  number  of  inde¬ 
pendent  variables  in  the  model.  Linear  regression  routines 
are  available  on  computerized  statistics  packages  and  are 
necessary  to  complete  the  analysis.  Regression  can  be  accom¬ 
plished  using  ordinal  or  nominal  variables;  however,  this 
requires  modifications  to  the  basic  model.  For  further 
references  for  applying  regression  see  (Ref  21:21-272),  and 
(Ref  20:34-208). 
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Objective:  Determine  the  relationship  between  radar  detec¬ 
tion  range  and  target  altitude. 

MOE:  radar  detection  range. 

Model : 

y  **  B  +  B.x,  +  e 
oil 

y:  radar  detection  range 

x^:  target  altitude  (in  100  x  feet) 

e:  random  or  unexplained  error 

Hypothesis : 

H  :  B  =  y  +  e 
o  o  z 

B1  =  0- 
Ha=  Bl  *  f 

Test: 

T  =  overall  F  -  from  printout 

W  «  F  .  (Ref  15:305) 

m,n, l-a^ 

m  =  regression  degrees  of  freedom  -  from  printout 
n  =  error  degrees  of  freedom  -  from  printout 

Acceptance  Region; 

T  <.  W 

If  B=1 :  y=p  x^=time 

If  B=2:  y=MOE  x^=independent  variables 

The  regression  technique  is  capable  of  determining  the 
relationship  with  more  than  one  independent  variable,  however 


the  graphical  relationship  is  not  easy  to  show.  Conclusions 
made  from  the  regression  results  should  be  restricted  to  the 
range  of  values  which  are  within  the  limits  of  the  data  col¬ 
lected.  Along  with  the  relationship  between  the  variables, 

—2 

the  explanatory  power  of  the  model  (R  )  should  be  reported 

and  can  be  interpreted  as  the  percent  of  variance  in  the 

dependent  variable  explained  by  the  model. 

BMD:  programs  -  P1R 

P2R 

(Ref  16:375-417) 

SPSS:  program  -  regression 

(Ref  22:370-367) 
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Lllliefors  Test 


The  Lilliefors  test  is  used  to  determine  if  a  sample 
could  have  come  from  a  normal  distribution.  Initially  the 
sample  must  be  ordered,  the  observed  cumulative  distribution 
at  is  equal  to  the  number  of  data  points  less  than  or 
equal  to  divided  by  n,  the  sample  size.  The  expected 
cumulative  frequency  is  found  in  standard  normal  tables 
after  standardizing  the  data.  The  test  statistic,  T,  is 
equal  to  the  maximum  difference,  for  any  x^,  between  P 
and  F  .  The  test  statistic,  T,  is  compared  to  the  Lillie- 
fors  test  statistic,  (Ref  10:357). 

Hypothesis : 

Hq:  The  sample  is  distributed  normal 

Ha:  The  sample  is  not  distributed  normal 


Test; 


x  _  number  of  x.£x. 

F'xi'obs  “  - J — i- 

n 


r<xi>exp  "  r  ^ 

<jiO 


(Ref  15:127) 


T  =  max  l»(x1)obs  -  F(xa) 
W  =  T 


n,  p 

n  =  sample  size 
p  =  1-a- 


exp  • 

(Table  III,  pp  A-101) 
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T  <.  W:  A=1  Return  to  @  pp.  A-38 
A=2  Return  to  (7)  PP*  A-38 


Example: 

An  analysis  of  component  reliability  uses  the  mean  time 
between  failures  (MTBF )  as  the  MOE.  Test  the  assumptions  of 
normality. 

Data  i 

Failure  23  ..  4  5  6  7  8  9  10  11 

Time  41.2  88.5  134.7  179.8  224.7  271.9  321.7  358.4  415.6  469.8 

TBF  41.2  47.3  46.2  45.1  44.9  47.2  49.8  36.7  57.2  54.2 

_  2 

y:TBF  y=47.0  s  =34.7  s=5.9  n=10 

Test; 

2 

H  :TBF  is  distributed  normal  with  u=47.0  and  o  =34.7. 
o 

Ordered  36.7  41.2  44.9  45.1  46.2  47.2  47.3  49.8  54.2  57.2 
Observations 

Observed  .1  .2  .3  .4  .5  .6  .7  .8  .9  1.0 

Cumulative 

Frequency 

F  (y_47. 0/5. 9)  .040  .163  .360  .373  .446  .514  .524  .683  .890  .958 
(Ref  15:127) 

T= | observed-Fj  .060  .037  .060  .027  .054  .086  .179  .117  .010  .042 
T  =  .179 

W  =  T  (Ref  10:463) 

n,  p 

n  =  10 
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'•It'  . 

nnni 


p  «  1  -  *1  =  *  9 
W  *  .239 

T  <  W  fail  to  reject  HQ  at  a2=.l,  therefore  there  is 
no  evidence  which  supports  the  claim  that  the 
data  is  not  normally  distributed  at  a2=.l. 
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ANOVA 


Analysis  of  variance  will  be  most  useful  to  compare 
more  than  two  means  simultaneously.  The  samples  which  are 
tested  can  be  defined  as  levels  of  factors  or  treatments, 
and  ANOVA  can  determine  which  factors  and/or  levels  repre¬ 
sent  significant  effects  on  the  overall  mean.  Experimental 
design  is  of  great  importance,  if  ANOVA  is  to  be  employed 
effectively.  For  a  complete  discussion  of  ANOVA  and  its 
design  see  \Ref  18)  and  (Ref  5:194-333). 

Example: 

Objective:  Determine  the  main  determinants  of  radar 
detection. 

MOE:  radar  detection  range 

Question:  Is  radar  detection  affected  either  positively 

or  negatively  by  aircraft  altitude  or  radar 
pulse  repetition  frequency  (PRF) 

Model : 

y  =  y+  A  +  P  +  e 

y:  radar  detection  range 

U :  the  overall  mean  detection  range 

A:  the  affect  of  altitude  on  y 

P;  the  affect  of  PRF  on  y 

e:  random  or  unexplained  error 

Hypothesis :  There  is  no  affect  due  to  altitude  or  PRF 
on  radar  detection  range. 
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Test : 

T  =  F  -  from  printout 
W  =  F  -,  (Ref  15:305) 

m  «  degrees  of  freedom  for  factors 
n  =  degrees  of  freedom  for  error 

Acceptance  Region 
T  <.  W 

This  ANOVA  model,  known  as  two-way,  can  be  used  to 
detect  the  effect -of  different  levels,  e.g.  high  or  low,  of 
each  factor  or  the  effect  of  interaction  between  the  factors, 
altitude  and  PRF.  Interaction  between  factors  is  the  effect 
of  combinations  of  factors  not  already  explained  by  the  fac¬ 
tors  alone  (e.g.  AP,  the  interaction  between  altitude  and 
PRF) .  Interaction  terms  should  not  be  included  in  the  model 
unless  the  interaction  can  be  interpreted.  In  general  in¬ 
teraction  beyond  two  terms  is  not  reported.  To  carry  out 
any  ANOVA  requires  at  least  one  observation  in  each  cell, 
however,  it  should  be  noted  that  interaction  cannot  be  tested 
without  replications  within  cells. 

The  ANOVA  technique  is  accomplished  by  comparisons 
between  row  and  column  means  and  the  overall  mean.  Simple 
ANOVA  can  be  accomplished  with  the  help  of  a  hand  calculator. 
However,  most  statistical  software  packages  have  ANOVA  rou¬ 
tines  which  will  make  the  job  easier.  It  should  be  noted 
that  these  computerized  routines  can  often  only  be  used  to 
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analyze  fixed  effect  models  without  some  manipulation  of  the 
output.  A  factor  is  fixed  if  the  levels  which  are  specified 
are  the  only  levels  of  interest  and  no  interpolations  or 
extrapolations  will  be  made.  Consult  the  references  already 
listed  for  the  procedure  to  be  applied  to  random  or  mixed  effect 
models. 

BMD  Programs  -  P1V 

P2V 

P3V 

(Ref  16:523-602) 

SPSS  Program  -  ANOVA 
(Ref  22:398-433) 

Go  To  Reporting  Results  -  C.  pp.  A-81 


Sian  Test 


The  sign  test  is  a  nonparametric  test  which  can  be  used 
to  test  one  average  against  a  hypothesized  value  or  to  com¬ 
pare  two  averages. 

Hypothesis :  Bring  hypothesis  (I)  from  flow  chart 
Test: 


C=1  (one 

sample) 

C=2  (two  samples) 

1=1,  1=3 

T=no.  of 
Q*+E  . 

1=2 

observations , 

x.,>  T=no.  of  x.  >  y.  +  E 

1  i  X 

T=no .  of 

observations , 

x. ,>Q*-E 

n  <_  20 

n  >  20 

U=n-H 

U=n/2  +  W  (  /n/4  ) 

L=W 

L-n/2  -  W  ( /n74  ) 

W=S 

n,  a 

(Ref  15:398) 

W=X:  F (X) =l-a 

(Ref  15:127) 

Acceptance  Reaion: 

1=1 

1=2 

1=3 

L<T<U 

L<T 

T<U 

Example:  Determine  whether  a  maintenance  task  is  easier  to 
accomplish  using  process  B  than  process  A.  Ten  maintenance 
personnel  are  asked  to  perform  the  task  using  each  process 
and  then  to  state  their  preference.  Assign  a  "+"  to  those 
preferring  process  A  and  a  to  those  who  prefer  process 


ft  it 


>  t  £  •  • 


Hq:  P  (+)  >  p(-) 


i.e.  process  A  is  preferred  to  process  B, 


Ha:  P(+)  <  P(-) 


Data: 


Three  people  preferred  process  A. 
Seven  people  preferred  process  B. 


n=lO 


a=2a^/2= . 1 


Test:  (two  sample)  (1=2) 


T  =  3 
L  =  1 


(Ref  15:398) 


L  £  T  fail  to  reject  HQ  at  =  .05 

There  is  no  evidence  to  support  a  claim 
that  process  B  is  preferred  to  process  A, 
Go  To  Reporting  Results  -  pp.  A-78 
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Mann-Whitnev  Test 


The  Mann-Whitney  test  is  a  nonparametric  test  for  com 


paring  the  average 

values  of  two  independent  samples  (Ref 

10:215-223) . 

• 

Sample  1 

Sample  2 

X11 

X21 

X12 

• 

X22 

• 

• 

• 

X, 

In 

• 

X. 

2m 

N=n+m 

• 

R  (xAj  )  :  the 

rank,  from  1  to  N,  assigned  to  each  obser 

votion  in  the  combined  samples.  A  rank  of  1 
is  assigned  to  the  smallest  observed  values 
and  N  to  the  largest,  tied  observations  should 
each  be  given  the  average  rank. 

Hypothesis :  bring  hypothesis  (I)  from  flow  chart 

Test: 

n 

T  «  l  R(x  ) 
i=l 

U  «  n (N+l) -W 

L  *  W 

W  =  U  (Ref  15:406) 

m ,  n ,  a 

Acceptance  Region: 

1=3 


L<T<U 


L<T 


Example : 


Determine  whether  the  average  detection  range  (nm)  of 
radar  A  is  different  from  that  of  radar  B,  a^=.l. 

% 

Hypothesis; 

Hq:  jE(A)-E(B) |£0,  i.e.  there  is  no  difference 

in  the  detection  ranges  of  radar  A  and  radar  B. 


Data:  (ordered) 

Radar  BBABBAABABAABABA 


Detection 
range _ 37 

Rank  1 


38  38  39  41  42  42  42 
2.5  2.5  4  5  7  7  7 


43  44  45  46  47  49  49  51 
9  10  11  12  13  14.5  14516 


m-8  n=8  N=16 


Tests ; 

T=2 . 5  +7+7+9+  11  +  12+  14.5  +  16  «  79 

W  =  U  (Ref  15:406) 

m,n,a 

W  *  19 

U  «=  8(16+1)  -  19  =  117 

L  =  19 

L<_T<_U  therefore  fail  to  reject  HQ  at  a^-l/  there 
is  no  evidence  to  support  a  claim  that  the 
detection  range  of  radar  A  is  significantly 
different  from  radar  B  at  a^= 

Go  To  Reporting  Results  -  B.  pp.  A-80 


nnci 
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Kruskall-Wsllis  Test 

The  Kruskall-Wallis  test  is  an  extension  of  the  Mann- 
Whitney  test  for  two  samples  to  more  than  two  samples  (Ref 
10:229).  The  test  is  used  to  test  the  hypothesis  that  k  in¬ 
dependent  populations  are  identical  against  the  alternative 
that  at  least  one  population  tends  to  yield  larger  values 
than  at  least  one  other  population.  The  samples  are  assumed 
to  be  independent;  that  is,  the  test  was  carried  out  so  that 
outcomes  in  separate  samples  are  unrelated.  If  this  is  not 
the  case  the  Friedman  Test  for  related  samples  should  be 
used  (pp.  A-72) . 


Sample  1 

Sample  2 

•  •  • 

Sample  k 

X11 

X21 

xkl 

x12 

X22 

Xk2 

• 

• 

• 

• 

• 

• 

• 

• 

x. 

lnl 

X2n2 

x, 

knk 

k 

N  =  E  n. 
i-1  X 

R(xij):  the  rank  from  1  to  N  assigned  to  the  combined 

sample  observations.  A  rank  of  1  is  assigned 
to  the  smallest  observation  and  N  to  the  largest, 
tied  observations  should  each  receive  the 
average  rank. 


R 


J 


Hypothesis : 

Hq:  E^)  =  E(x2)  =  ...  E(X^) 

The  k  populations  are  identically  distributed. 
H^:  At  least  one  of  the  populations  tends  to  yield 
larger  observations  than  at  least  one  of  the 
others. 


Test: 

The  test  statistic  T  is  calculated  as: 


S 


2 


k 

E 

i=l 


ni  o 

I  R  (x .  . )  2 
j=l 


N  (N+l)  2 
4 


'e  ^  -MMli 

i=l  nL 


W  = 


,  F 


(Ref  15:294) 


n  =  k-1 


F  «  1— a 


1 


Acceptance  Region: 


T  <  W 


Go  To  Reporting  Results  -  C.  pp.  A-81 
Pairwise  Comparison: 

If,  and  only  if,  the  Hq  is  rejected  pairwise  com¬ 
parisons  can  be  made  to  detect  individual  differences  between 
two  samples. 
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SSSSSSSS 


A  H  -o!  « ‘MV '  !T  va.13  vrr ',T.'.vvLcrvrJCLT.\xr. u,  ~v  ,^r. 


•  ."VT~*  t~*  v'w  i-m  \n  *n*  ya  «- 


TX  =  i!i  .  !Li| 

ni  nj 


w,  =  (t  )  (s2  N-IjtlT)  **  (“•  +  J-  )** 

1  n,F  N_k  nL  n ^ 


t  '  -  from  (Ref  15:283) 
n  #  i? 


n  «  N-k 
F 


1  -  a j,/2 


A  significant  difference  exists  if  T^>W^ 

Example: 

Determine  if  there  is  a  significant  difference  in  the 
results  of  air  combat  using  three  different  tactics,  the  MOE 
is  average  exchange  ratio,  (ER=blue  force  "killed"/red  force 
“killed"),  a^.l. 


Hypothesis: 


Hq:  The  expected  outcome  of  tactic  A  (E (A) )  is 

equal  to  the  expected  outcome  of  tactics  B 
and  C,  i.e.  E (A)  =  £(B)  =  E(C) 


Data: 


Tactic  A 


Tactic  B 


Tactic  C 


ER 

Rank 

ER 

Rank 

ER 

Rank 

.20 

1 

.33 

5.5 

.40 

0.5 

.25 

2 

.33 

5.5 

.50 

12 

.30 

3 

.40 

8.5 

.60 

15.5 

.33 

5.5 

.50 

12 

.60 

15.5 

.33 

5.5 

.50 

12 

.67 

17 

.50 

12 

.50 

12 

.75 

18 

29 

■5375 

863 
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N  =  18 


S2  =  ~“j-  (l2+22+32+4  (5.52)+2  (8.52}+5(122)+2(15.52)+172+182)  • 

18  (19)  2 
"4 

=  27.559 


27.559 


r292  55. 52  A  86. 52  18  (19] 

1  6  +  6  +  6  '  ~  4 


=  10.01 


W  =  X 


k-l,l-an 


k  =  3 
W  =  4.61 

T  >  W,  therefore  reject  Hq  at  a^=.l,  there  is  evidence 
that  exchange  rate  changes  with  tactics  at  a^=.l. 

There  is  a  10*  (a^)  chance  the  difference  detected  is 
due  to  random  chance. 

Determine  which  tactics  of  the  three  differ. 

A  vs .  B 

T!  =  "  "6  I  =  4'417 

W.  =  1.753  (27.559  (18-1-10.01)  )**(4  + 

1  18-3  6  6 

=  3.627 

T1  >  W1  therefore  tactic  A  and  B  are  significantly 


different, 


B  vs.  C 


T1  =  1^6^  "  ^6^1  =  5*167 
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affc.  ^  ty-L. .  i 


^  »fcn im  -  ^ 


Wx  =-  3.627 

T1  >  N1  therefore  tactic  B  and  C  are  significantly 
different. 


A  vs.  _C__ 

T  c  |29 

T1  I  6  "  6  » 


9.583 


W1  c  3.627 

T1  >  W1  therefore  tactic  A  and  C  are  significantly 
different. 
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Friadman  Test 


The  Friedman  test,  like  the  Kruskall-Wallis  test,  is 
used  to  determine  which  treatments,  if  any,  in  an  experiment 
cause  significantly  different  measured  values  of  the  response 
variable  or  MOE.  The  Friedman  test  is  used  for  related  sam¬ 
ples.  that  is,  samples  which  lack  independence.  The  samples 
are  considered  to  be  related  when  the  test  is  carried  out  in 
such  a  way  that  more  than  one  sample  is  measured  at  the  same 
time,  or  some  other  factor  causes  the  relationship  (i.e. 
same  pilot)  that  is  some  uncontrolled  factor  may  influence 
measurements  between  samples  (Ref  10:299-308). 


Treatment 

1  2  ...  k 


Treatment:  The  factor  which  is  to  be  tested,  with  k  levels. 

Block:  The  environmental  conditions  which  are  controlled 

in  the  experiment,  (e.g.  aircraft  tail  number, 
trial)  . 

Mx^j);  The  rank  from  1  to  k  which  is  assigned  to  each 
x^j  within  a  block,  a  rank  of  1  is  assigned  to 
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the  smallest  observation  and  k  to  the  largest. 

In  case  of  a  tie  assign  the  average  rank  to  both 
observations, 
b 

R.  =  l  R (x  ,  . ) 

J  1=1 


The  blocks  arc  ranked  from  1  to  b  according  to  the  range 
of  values  within  a  block  (maximum  x^j  -  minimum  x^).  Rank  1 
is  assigned  to  the  block  with  the  smallest  range. 

=  the  rank  assigned  to  block  i. 


Qi  (R<xij) 


i=l 


k±l) 

2 


k:  The  number  of  treatments 
b:  The  number  of  blocks 


Hvtx>thesis ; 


Hq:  The  treatments  have  equal  effects  on  the  response 
variable  or  MOE. 

H^:  At  least  one  of  the  treatments  tends  to  yield 

larger  observations  than  at  least  one  other  treat 
ment. 


Test: 

A 

T 


b 

l 

i=l 


(b-lj  B 
A-B 


2 
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(Ref  15:305) 


m  =  b-1 
n  =  k-1 

Acceptance  Region: 

T  <.  W 

Go  To  Reporting  Results  -  C.  pp.  A-81 
Example: 

Determine  which  radar  modification  results  in  the  long¬ 
est  radar  detection  range;  all  trials  involve  three  aircraft 
simultaneously,  one  with  each  modification,  a^=.l. 

Hypothesis : 

Hq:  Modification  A,  B,  and  C  are  equally  effective 
in  terms  of  their  effect  on  detection  range. 

Data: 


Modification 

Trial 

2l_ 

rank 

B 

rank 

C 

rank 

1 

23 

3 

22 

1.5 

22 

1.5 

2 

27 

2 

28 

3 

26 

1 

3 

24 

1.5 

24 

1.5 

27 

3 

4 

22 

1 

24 

2 

26 

3 

5 

23 

1 

25 

2 

26 

3 

6 

21 

1 

22 

2 

24 

3 

7 

29 

1 

30 

2 

33 

3 

8 

28 

_ 2 

12.5 

27 

_ 1 

15 

31 

_ 3 

20.5 

A-74 


•  3  c  “I 


Test : 


A  =  8  ( 3 2 )  +  6  (22)  -»•  4(1. 52)  +  6(12)  =  111 

B  *  1  (12. 52  +  152  +  20. 52)  =  100.188 
8 

T  =  (8-1) (100.188)  =  64.86 
(111-100.188) 

W  =  F  . 

m  /  n  / 1  ^ 

m  *=  b-1  «=  7 
n  =  k-1  -  2 

W  =  9.35,  (Ref  15:305) 

T  >  W  therefore  reject  HQ  at  a^a.l,  there  is 
evidence  that  the  modifications  to  the 
.  radar  do  not  result  in  equally  effective 
radar  perf ormance . 
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Sieqel-Tukev  Test. 


The  siegel-Tukey  test  is  performed  in  a  manner  very 
similar  to  the  Mann-Whitney  test  and  uses  identical  statis¬ 
tical  tables.  The  test  compares  the  equality,  or  inequality, 
of  the  variances  from  two  populations.  When  using  this  test 
the  sample  variance  should  be  computed  using  the  radial  miss 
distance  (RMD)  from  the  MPI  (mean  point  of  impact) ,  (RMD^  = 


Y<  > 


Sample  1 
X11 


Sample  2 
X21 


12 


"22 


xln  X2m 

N  =  n+m 

The  two  samples  should  be  combined  and  ordered  as  for 
the  Mann-Whitney  test,  however,  the  ranks,  R  (xij )  should 
be  assigned  so  that  rank  1  is  given  to  the  smallest  value, 
rank  2  to  the  largest,  rank  3  to  the  second  largest,  rank  4 
to  the  second  smallest,  rank  5  to  the  third  smallest,  and 
so  on.  In  case  of  a  tie,  assign  the  average  rank  to  each 
observation. 


Hypothesis; 


1) 


_2  _  2 
°1  "  °2 


°1  *  °2 


two-tail  test  a=aj/2 
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2) 


H  : 
O 


2 

1 


one-tail  test  a=a^ 


Test : 

n 

T  =  I  R(x.  .) 

j=i  lj 

U  =  n (N+l)  -  W 
L  =  W 

W  =  U  „  (Ref  15:406) 

in,n,a 


Acceptance  Region: 


Hypothesis 

1  ,  L  <.  T  £  U 

2  L  <.  T 

Go  To  Reporting  Results  -  pp.  A-78 


Example : 

Determine  whether  delivery  accuracy  of  iron  bombs  (MK-82) 
using  fire  control  computer  mode  A  is  better  than  the  accur¬ 
acy  when  using  mode  B,  a^=.l. 

Hypothesis: 


The  variance  around  the 


greater  or  equal  to  the 

.  2  2. 

°A  ^  °B> 


MPI  using  mode  A  is 
variance  using  mode  B. 


r.' 


A-77 


Data : 


Mode 


AABAABABABABBB 


Radial  miss 

distance  (ft.)  7  8  11  13  14  15  15  22  24  27  28  28  35  39 
Rank  14589  12.5  12.5  14  11  10  6.5  6.5  3  2 

T  =  (1  +  4  +  8  +  9  +  12.5  +  11  +  6.5) 

*  52 

V  -  U  „  (Ref  15:406) 

m  #  ti|  d 

m  -  7 
n  =  7 
a  =  .1 
W  =  13 
L  =  13 

L  <.  T  therefore  fail  to  reject  Hq  at  a^=.l,  there 
is  no  evidence  to  support  the  claim  that  the 
variance  of  mode  A  is  less  than  the  variance 
of  mode  B. 

Reporting  Results 

A.  One  Sample  Tests  -  Fill  in  parenthesis  for  the  correct 

combination  of  H  and  test  result. 

o 

V* 

p=p* 

|u-Vi*  |  1  E 
|E(x)  -  Q*|  <  E 
ZEP  =  Q* 
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I  u\ 


u\ 


Reject  H  :  "Reject  H  at  ( (1-a.  )  xlOO)  %  confidence 
level.  There  is  evidence  that  (MOE)  is  significantly 
different  from  (hypothesized  value,  e.g.  Q*).  The  pro¬ 
bability  that  the  difference  detected  is  due  purely* to 
randomness  is  equal  to  (a^)." 

Fail  to  Reject  H  :  "Fail  to  reject  H  at  ((1-a. )x 

_ _ _ o  O  X 

100)%  confidence  level.  There  is  insufficient  evidence 
to  indicate  that  (MOE)  is  significantly  different  from 
(hypothesized  value) ." 

H  : 
o 

P  >.  P* 
y  >.  p*  -  E 

E(x)  2.  Q*  -  E 
ZEP  >  Q* 

Reject  Hq:  "Reject  Hq  at  ( (l-a^)xlOO) X  confidence 
level.  There  is  evidence  that  (MOE)  is  significantly 
less  than  (hypothesized  value) .  The  probability  that 
the  difference  detected  is  due  purely  to  randomness  is 
equal  to  (a^)." 

Fail  to  Reject  Hq:  "Fail  to  reject  Hq  at  ((l-a^)x 
100)%  confidence  level.  There  is  insufficient  evidence 
to  indicate  that  (MOE)  is  significantly  less  than 
(hypothesized  value)." 
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p  <  p* 
vi  <  V*  +  E 
E(x)  <  Q*  +  E 
2EP  <  Q* 

Reject  HQ :  "Reject  HQ  at  ( (l-a^)xlOO) *  confidence 
level.  There  is  evidence  that  (MOE  is  significantly 
greater  than  (hypothesized  value).  The  probability 
that  the  difference  detected  is  due  purely  to  randomness 
is  equal  to  (a^)." 

Fail  to  reject  HQ:  "Fail  to  reject  HQ  at  ((l-a^) 
xl00)%  confidence  level.  There  is  insufficient  evidence 
that  (MOE)  is  significantly  greater  than  (hypothesised 
value) . " 

B.  Two  Samples  -  Fill  in  parenthesis  for  correct  combination 

of  H  and  test  results, 
o 

H  : 
o 

|vix  -  U2I  <  E 

|  E  (x)  -  E(y)  I  <  E 
ZEPX  •  ZEPj 

Pi  -  P2 

Reject  H  j  "Reject  H  at  ( (1-a, ) xlOO) %  confidence 
o  o  i 

level.  There  is  evidence  that  (MOE^)  is  significantly 
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different  from  (MOE2) .  The  probability  that  the  dif¬ 
ference  detected  is  due  purely  to  randomness  is  equal 
to  (a1)." 

Fail  to  reject  Hq:  "Fail  to  reject  Hq  at 
( (l-a1)xlOO) %  confidence  level.  There  is  insufficient 
evidence  to  indicate  that  (MOE-^)  is  significantly 
different  from  (MOE^)." 

H  : 
o 

"l  i-  •*2'+  E 
E  (x)  <_  E  (y)  +  E 

Reject  Hq:  "Reject  HQ  at  (  (l-a^JxlOO) %  confi¬ 
dence  level.  There  is  evidence  that  (MOE^)  is  greater 
than  (MOE^) .  The  probability  that  the  difference 
detected  is  due  purely  to  randomness  is  equal  to  (a^)." 

Fail  to  reject  H  :  "Fail  to  reject  H  at 
_ _ _ o  o 

( (l-a^)xlOO) %  confidence  level.  There  is  insufficient 
evidence  to  indicate  that  (MOE-^)  is  significantly 
greater  than  (MOE2)." 

C.  More  Than  Two  Samples  -  Fill  in  parenthesis. 

Reject  Hq;  "Reject  Hc  at  { (l-a-^xlOO)  X  confidence 
level.  There  is  evidence  that  at  least  one  of  the  treat¬ 
ments  tends  to  yield  larger  values  of  (MOE)  than  at  least 
one  other  treatment.  The  probability  that  this  result 
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is  due  purely  to  randomness  is  equal  to  (a^)." 

Fail  to  reject  H  :  "Fail  to  reject  H  at  ( (1-a, ) 

_ _ o  o  1 

xlOO)X  confidence  level.  There  is  insufficient  evidence 
to  indicate  that  any  of  the  treatments  yield  larger 
values  of  (MOE)  than  any  other  treatment." 


Using  the  Guide 

An  Example;  Using  the  following  data  determine  whether 
a  new  avionics  component  (B)  is  more  reliable  than  the  cur¬ 
rently  used  component  (A) .  The  data  which  has  been  collected 
is  the  time  between  failure,  the  selected  MOE  is  MTBF  (mean 
time  between  failure) . 


The  Data: 


Component  Failure  12345678910s 
Time  of  failure 

Component  A  2.6  3.7  2.9  2.8  3.3  4.1  3.6  4.2  3.0  3.1  .55 
Component  B  4.2  3.3  3.8  4.2  3.1  2.9  4.4  3.7  3.9  3.5  .50 


STEP  Is 

time 
between 
failure 

failure 

Figure  A-2.  Time  Between  Failures  vs  Failure 
No  trends  are  detected. 
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E  - 


1  hours 


r.V.’N  %'  x  -  %  -  « 


SA(VSB>  (t.9.J2 


.5  (1)  (1.282/.!)*  =  84 


STEP  II: 


»  .1 

a2  **  .1 


STEP  III: 


Hq:  Sample  A  is  a  random  sample. 

H^:  Sample  A  is  not  a  random  sample. 


XA:  MTBF  of  sample  A. 

XA  =  3.33 

ni  =  4  n2  =  6 

Failure:  123456789  10 


TBF (X±) :  2.6  3.7  2.9  2.8  3.3  4.1  3.6  4.2  3.0  3.1 


Xi>X: 
runs : 


+  +  + 


1  2 


T  =  5  a2  -  .1 
L  =  U4,6,.05  =  3 
U  =  U4,6,.95  =  8 
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L  <  T  <  U  therefore  fail  to  reject  H  ,  there  is 
—  —  o 

no  evidence  that  sample  A  is  not  a 
random  sample. 

Hq:  Sample  B  is  a  random  sample. 

H  :  Sample  r  is  not  a  random  sample. 

A 

XB  =  3.7 

n1  =  5  n2  =  5 


Failure:  1 

TBF  (Xi):4.2 
Xi>X:  + 

runs :  1 


2  3  4 

3.3  3.8  4.2 

+  + 

2  3 


5  6  7 

3.1  2.9  4.4 

-  -  + 

4  5 


8  9  10 

3.7  3.4  3.5 

+ 

6  7  8 


T  =  8  a2=.l 

L  =  U5.S..05  *  3 


U  ■  US,5,.95  *  8 


L  <  T  <  U 


therefore  fail  to  reject  Hq/  there  is 
no  evidence  that  sample  B  is  not  a 
random  sample. 


Type  of  MOE?  -  Average 


34 

nA  -  N^A? 

-  No. 

34 

nB  - 

-  No 

Go  To  pp.  A-4  5 _ fT)  Nonparametric  Tests 


One  sample?  -  No 
Two  samples?  -  'Yes 


H  :  E  (B)  <.  E  (A) 
o 

H  s  E  (B)  >  E  (A) 

A 

a  =  -1  1=3 

Paired  samples?  -  No 

no  To  do.  A-65 _ Mann -Whitney. 

Sampl e  A 

Sample 

:  B 

Si 

hi 

Si 

hs 

2.6 

1 

4.2 

18 

3.7 

12.5 

3.3 

8.5 

2.9 

3.5 

3.8 

14 

2.8 

2 

4.2 

18 

3.3 

8.5 

3.1 

6.5 

4.1 

16 

2.9 

3.5 

3.6 

11 

4.4 

20 

4.2 

18 

3.7 

12.5 

3.0 

5 

3.9 

15 

3.1 

6.5 

3.5 

10 

84 

m  = 

n  =  10  N  = 

20 

10 

T  = 

E  R  ,  -  84 
j=l  J 

L  = 

U  n  ,  =  32 

U  = 

10(20+1)  -  32 

=  178 

1=3 

r.  <  T  therefore  fail  to  reject  H 


at  a=.l 
o 


Fail  to  reject  HQ  at  90*  confidence  level.  There  is 
insufficient  evidence  to  indicate  that  the  MTBF  of  the  new 
component  is  greater  than  that  of  the  old  component. 


VIII.  REFERENCES 

In  order  to  write  a  truly  complete  guide  for  OT&E  data 
analysis,  it  would  be  necessary  to  include  large  sections  of 
statistical  theory  and  tables  of  probability  distribution 
functions.  This  has  not  been  done  in  the  interest  of  utility, 
that  is,  so  that  this  guide  will  be  used  as  a  quick  refer¬ 
ence,  not  as  a  complete  treatment  of  the  subject  matter. 
Statistical  tables,  texts  and  software  guides  may  also  be 
necessary  to  perform  the  statistical  analysis  suggested  in 
this  guide.  In  order  to  standardize  as  much  as  possible, 
all  of  the  tables  referred  to  in  the  guide  can  be  found  in 
the  CRC  Handbook  for' Probability  and  Statistics  (Ref  15).  The 
exceptions  are  the  binomial  tables  which  are  found  in  Table  II 
and  the  Lilliefors  test  statistic  in  Table  III.  All  of  the 
texts  cited  in  the  bibliography  contain  statistical  tables,* 
however,  none  are  as  complete  as  the  CRC  (Ref  15)  . 

The  textbooks  which  are  referenced  in  the  guide  are 
listed  in  the  bibliography  (pp.  A-102)  by  topic  in  increasing 
order  of  mathematical  complexity  within  each  topic.  Consult¬ 
ing  a  text  prior  to  using  an  unfamiliar  statistical  technique 
will  result  in  a  better  understanding  by  the  user.  At  least 
two  statistical  references  will  be  necessary  as  a  minimum, 
one  in  the  area  of  general  statistics  and  one  in  nonpara- 
metric  statistics. 

Two  statistical  software  packages,  SPSS  (Ref  22)  and  BMD 
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(Ref  16)  neve  been  cited  in  this  guide  because  of  their 
availability  to  the  Air  Force  analyst  and  the  quality  of 
their  documentation.  SPSS  and  BMD  are  available  at  many 
bases  and  the  copies  can  be  transferred  within  the  Air  Force. 
Documentation  can  be  ordered  from: 

BMD 

University  of  California  Press 

2223  Fulton  Street 

Berkeley,  California  94720 

SPSS 

SPSS  Incorporate  Inc. 

Suite  3300 

44  North  Michigan  Avenue 

Chicago,  Illinois  60611 

As  important  to  the  OT&E  analyst  as  textbooks  and  soft¬ 
ware  are  contacts  within  the  Air  Force  with  OT&E  and/or 
statistical  experience.  The  following  is  only  a  partial 
list  of  offices  where  personnel  with  these  backgrounds  can 
be  consulted: 

Air  Force  Test  and  Evaluation  Center 

AFTEC/OA  AV-244-04  37 

Air  Force  Institute  of  Technology 

AFIT/Math  Department  AV-285-3098 

Operational  Sciences  Dept.  AV-285-2549 

U.S.  Air  Force  Academy 

USAFA/Math  Department  AV-259-4470 


Table  IX  Blnonlal  Distribution* 


a 

y 

p».05 

.10 

.15 

.20 

.25 

.30 

.35 

.40 

.45 

1 

0 

.9500 

.9000 

.8500 

.8000 

.7500 

.7000 

.5400 

.6000 

.5500 

i 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

2 

0 

.9025 

.8100 

.7225 

.6400 

.5625 

.4900 

.4225 

.3600 

.3025 

l 

.9975 

.9900 

.9775 

.9600 

.9375 

.9100 

.8775 

.8400 

.7975 

2 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

3 

0 

.8574 

.7290 

.6141 

.5120 

.4n9 

.3430 

.2746 

.2160 

.1664 

1 

.9928 

.9720 

.9392 

.8960 

.84  38 

.7840 

.7182 

.64  80 

.5748 

2 

.9999 

.9990 

,9966 

.9920 

.9844 

.9730 

.9571 

.9360 

.9089 

3 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

4 

0 

.8145 

.6561 

.5220 

.4096 

.3164 

.2401 

.1785 

.1296 

.0915 

1 

.9860 

.9477 

.8905 

.8192 

.7383 

.6517 

.5630 

.4752 

.3910 

2 

.9995 

.9963 

.9880 

.9728 

.9492 

.9163 

.8735 

.8208 

.7585 

3 

1.0000 

.9999 

.9995 

.9984 

.9961 

.9919 

.9850 

.9743 

.9590 

4 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

S 

0 

.7738 

.5905* 

.4437 

.3277 

.2373 

.1681 

.1160 

.0778 

.0503 

1 

.9774 

.9185 

.8352 

.7373 

.6328 

.5282 

.4284 

.3370 

.2562 

2 

.9988 

.9914 

.9734 

.94a 

.8965 

.8369 

.7648 

.6826 

.5931 

3 

1.0000 

.9995 

.9978 

.9933 

.9844 

.9692 

.9460 

.9130 

.8688 

4 

1.0000 

1.0000 

.9999 

.9997 

.9990 

.9976 

.9947 

.9898 

.9815 

5 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

6 

0 

.7351 

.5314 

13771 

.2621 

.1780 

.1176 

.0754 

.0567 

.0277 

1 

.9672 

.8857 

.7765 

.6554 

.5339 

.4202 

.3191 

.2333 

.1636 

2 

.9978 

.9842 

.9527 

.9011 

.8306 

.7443 

.6471 

.5443 

.4415 

3 

.9999 

.9987 

.9941 

.9830 

.9624 

.9295 

.8826 

.8208 

.7447 

4 

1.0000 

.9999 

.9996 

.9984 

.9954 

.9891 

.9777 

.9590 

.9308 

5 

1.0000 

1.0000 

1.0000 

.9999 

.9998 

.9993 

.9982 

.9959 

.9917 

6 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

7 

0 

.6983 

.4783 

.3206 

.2097 

.1335 

.0824 

.0490 

.0280 

.0152 

1 

.9556 

.8503 

.7166 

,5767 

.4449 

.3294 

.2338 

.1586 

2 

.9962 

.9743 

.9262 

.8520 

.7564 

.6471 

.5323 

.4199 

.3164 

3 

.9998 

.9973 

.9879 

.9667 

.9294 

.8740 

.8002 

.7102 

.6083 

4 

1.0000 

.9998 

.9988 

.9953 

.9871 

.9712 

.9444 

.9037 

.8471 

5 

1.0000 

1.0000 

.9999 

.9996 

.9987 

.9962 

.9910 

.9812 

.9643 

ft 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9998 

.9994 

.9984 

.9963 

7 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

*Y  has  the  blnonlal  distribution  with  paraaeters  n  and  p.  The  entries  are  the  values 
o £  P(Y<y)-^jt0(“)p1'(l-p)n”1,  for  p  ranging  from  .05  to  .95. 
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• 

7 

p*.30 

.53 

.60 

.65 

.70 

.75 

.80 

.85 

.90 

.95 

1 

0 

.3000 

.4300 

.4000 

.3300 

.3000 

.2500 

.2000 

.1500 

.1000 

.0500 

1 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

2 

0 

.2300 

.2023 

.1600 

.1225 

.0900 

.0625 

.0400 

.0225 

.0100 

.0025 

1 

.7500 

.6973 

.6400 

.5775 

.5100 

.4375 

.3600 

.2775 

.1900 

.0975 

2 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

3 

0 

.1250 

.0911 

.0640 

.0429 

.0270 

.0156 

.0080 

.0034 

.0010 

.0001 

1 

.5000 

.4252 

.3520 

.2818 

.2160 

.1562 

.1040 

.0608 

.0280 

.0072 

2 

.8750 

.8336 

.7840 

.7254 

.6570 

.5781 

.4880 

.3859 

.2710 

.1426 

3 

1.00C0 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

4 

0 

.0625 

.0410 

.0256 

.0150 

.0081 

.0039 

.0016 

.0005 

..  .0001 

.0000 

1 

.3125 

.2415 

.1792 

.126S 

.0837 

.0508 

.0272 

.0120 

.0037 

.0005 

2 

.6875 

.6090 

.5248 

.4370 

.3484 

.2617 

.1808 

.1095 

.0523 

.0140 

3 

.9375 

.9085 

.8704 

.8215 

.7599 

.6836 

.5904 

.4780 

.3439 

.1855 

4 

1.0000 

1.0000 

.1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

S 

0 

.0312 

.0185 

.0102 

.0053 

.0024 

.0010 

.0003 

.0001 

.0000 

.0000 

l 

.1875 

.1312 

.0870 

.0540 

.0308 

.0156 

.0067 

.0022 

.0005 

.0000 

2 

.5000 

.4069 

.3174 

.2352 

.1631 

.1035 

.0579 

.0266 

.0086 

.0012 

3 

.8125 

.7438 

.6630 

.5716 

.4718 

.3672 

.2627 

.1648 

.0815 

.0226 

4 

.9688 

.9497 

.9222 

.8840 

.8319 

.7627 

.6723 

.5563 

.4095 

.2262 

5 

l.OOGO 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

6 

0 

.0156 

.0083 

.0041 

.0018 

.0007 

.0002 

.0001 

.0000 

.0000 

.0000 

1 

.1094 

.0692 

.0410 

.0223 

.0109 

.0046 

.0016 

.0004 

.0001 

.0000 

2 

.3438 

.2553 

.1792 

.1174 

.0705 

.0376 

.0170 

.0059 

.0013 

.0001 

3 

.6562 

.5585 

.4557 

.3529 

.2557 

.1694 

.0989 

.0473 

.0158 

.0022 

4 

.8906 

.8364 

.7667 

.6809 

.5798 

.4661 

.344  6 

.2235 

.1143 

.0328 

3 

.9844 

.9723 

.9533 

.9246 

.8824 

.8220 

.7379 

.6229 

.4686 

.2649 

4 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

7 

0 

.0078 

.0037 

.0016 

.0006 

.0002 

.0001 

.0000 

.0000 

.0000 

.0000 

1 

.0625 

.0357 

.0188 

.0090 

.0038 

.0013 

.0004 

.0001 

.0000 

.0000 

2 

.2266 

.1529 

.0963 

.0556 

.0288 

.0129 

.0047 

.0012 

.0002 

.0000 

3 

.5000 

.3917 

.2898 

.1998 

.1260 

.0706 

.0333 

.0121 

.0027 

.0002 

4 

.7734 

.6836 

.5801 

.4677 

.3529 

.2436 

.1480 

.0738 

.0257 

.0038 

S 

.9375 

.8976 

.8414 

.7662 

.6706 

.5551 

.4233 

.2834 

.1497 

.0444 

6 

.9322 

.9848 

.9720 

.9510 

.9176 

.8665 

.7903 

.6794 

.5217 

.3017 

7 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 
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a 

y 

•A 

o 

■* 

o. 

.10 

.15 

.20 

.25 

.30 

.35 

.40 

.45 

8 

0 

.6634 

.4305 

.2725 

.1678 

.1001 

.0576 

.0319 

.0168 

.0084 

l 

.9428 

.8131 

.6572 

.5033 

.3671 

.2553 

.1691 

.1064 

.0632 

2 

.9942 

.9619 

.8948 

.7969 

.6785 

.5518 

.4278 

.3154 

.2201 

3 

.9996 

.9950 

.9786 

.9437 

.8862 

.8059 

.7064 

.5941 

.4770 

4 

1.0000 

.9996 

.9971 

.9896 

.9727 

.9420 

.8939 

.8263 

.7396 

5 

1.0000 

1.0000 

.9998 

.9988 

.9958 

.9887 

.9747 

.9502 

.9115 

6 

1.0000 

1.0000 

1.0000 

.9999 

.9996 

.9987 

.9964 

.9915 

.9819 

7 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9988 

.9993 

.9983 

8 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

9 

0 

.6302 

.3874 

.2316 

.1342 

.0751 

.0404 

.0207 

.0101 

.0046 

1 

.9208 

.7748 

.5995 

.4362 

.3003 

.1960 

.1211 

.0705 

.0385 

2 

.9916 

.9470 

.8591 

.7382 

.6007 

.4628 

.3373 

.2318  - 

.1495 

3 

.9994 

.9917 

.9661 

.9144 

.8343 

.7297 

.6089 

.4826 

.3614 

4 

1.0000 

.9991 

.9944 

.9804 

.9511 

.9012 

.8283 

.7334 

.6214 

5 

1.0000 

.9999 

.9994 

.9969 

.9900 

.9747 

.9464 

.9006 

.8342 

6 

1.0000 

1.0000 

.  1.0000 

.9997 

.9987 

.9957 

.9888 

.9750 

.9502 

7 

1.0000 

1.0000 

.  1.0000 

1.0000 

.9999 

.9996 

.9986 

.9962 

.9909 

.  8 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9997 

.9992 

9 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

10 

0 

.5987 

.3487 

.1969 

.1074 

.0563 

.0282 

.0135 

.0060 

.0025 

1 

.9139 

.7361 

.5443 

.3758 

.2440 

.1493 

.0860 

.0464 

.0233 

2 

.9885 

.9298 

.8202 

.6778 

.5256 

.3828 

.2616 

.1673 

.0996 

3 

.9990 

.9872 

.9500 

.8791 

.7759 

.6496 

.5138 

.3823 

.2660 

4 

.9999 

.9984 

.9901 

.9672 

.9219 

.8497 

.7515 

.6331 

.5044 

5 

1.0000 

.9999 

.9986 

.9936 

.9803 

.9527 

.9051 

.8338 

.7384 

6 

1.0000 

1.0000 

.9999 

.9991 

.9965 

.9894 

.9740 

.9452 

.8980 

7 

1.0000 

1.0000 

1.0000 

.9999 

.9996 

.9984 

.9952 

.9877 

.9726 

8 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9995 

.9983 

.9955 

9 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9997 

10 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

11 

0 

.5688 

.3138 

.1673 

.0859 

.0422 

.0198 

.0088 

.0036 

.0014 

.  1 

.8981 

.6974 

.4922 

.3221 

.1971 

.1130 

.0606 

.0302 

.0139 

2 

.9848 

.9104 

.7788 

.6174 

.4552 

.3127 

.2001 

.1189 

.0652 

3 

.9984 

.9815 

.9306 

.8389 

.7133 

.5696 

.4256 

.2963 

.1911 

4 

.9999 

.9972 

.9841 

.9496 

.8854 

.7897 

.6683 

.5328 

.3971 

5 

1.0000 

.9997 

.9973 

.9883 

.9657 

.9218 

.8513 

.7535 

.6331 

6 

1.0000 

1.0000 

.9997 

.9980 

.9924 

.9784 

.9499 

.9006 

.8262 

7 

1.0000 

1.0000 

1.0000 

.9998 

.9988 

.9957 

.9878 

.9707 

.9390 

8 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9994 

.9980 

.9941 

.9852 

9 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9998 

.9993 

.9978 

10 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9998 

11 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 
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P«.S0 

.55 

.60 

.65 

.70 

.75 

.00 

.03 

.90 

.93 

0 

.0039 

.0017 

.0007 

.0002 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0332 

.0101 

.0003 

.0036 

.0013 

.0004 

.0001 

.0000 

.0000 

.0000 

.1(43 

.0005 

.0496 

.0233 

.0113 

.0042 

.0012 

.0002 

.0000  ' 

.0000 

.  3433 

.2604 

.1737 

.1061 

.0560 

.0273 

.0104 

.0029 

.0004 

.0000 

.6367 

.3230 

.4039 

.2936 

.1941 

.1136 

.0363 

.0214 

.0050 

.0004 

.0333 

.7799 

.6046 

.3722 

.4462 

.3215 

.2031 

.1032 

.0361 

.0036 

.9646 

.9360 

.6936 

.0309 

.7447 

.6329 

.4967 

.34  26 

.1669 

.0372 

.9961 

.9916 

.9032 

.9661 

.9424 

.0999 

.6322 

.7275 

.3693 

.3366 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.0020 

.0006 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0193 

.0091 

.0036 

.0014 

.0004 

.0001 

.0000 

.0000 

.0000 

.0000 

.0090 

.0496 

.0250 

.0112 

.0043 

.0013 

.0003 

.0000 

-  .0000 

.0000 

.2539 

.1630 

.0994 

.0536 

.0253 

.0100 

.0031 

.0006 

.0001 

.0000 

.3000 

.3706 

.2666 

.1717 

.0966 

.0469 

.0196 

.0056 

.0009 

.0000 

.7461 

.6306 

.3174 

.3911 

.2703 

.1637 

.0636 

.0339 

.0083 

.0006 

.9102 

.0303 

.7602 

.6627 

.5372 

.3993 

.2616 

.1409 

.0330 

.0004 

.9003 

.9613 

-.9293 

.0769 

.6040 

.6997 

.3636 

.4003 

.2232 

.0712 

.9960 

.9954 

.9699 

.9793 

.9396 

.9249 

.0636 

.7664 

.6126 

.3698 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.0010 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0107 

.0043 

.0017 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0347 

.0274 

.0123 

.0046 

.0016 

.0004 

.0001 

.0000 

.0000 

.0000 

.1719 

.1020 

.0340 

.0260 

.0106 

.0035 

.0009 

.0001 

.0000 

.0000 

.3770 

.2616 

.1662 

.0949 

.0473 

.0197 

.0064 

.0014 

.0001 

.0000 

.6230 

.4936 

.3669 

.2465 

.1303 

.0761 

.0326 

.0099 

.0016 

.0001 

.1201 

.7340 

.6177 

.4662 

.3304 

.2241 

.1209 

.0300 

.0126 

.0010 

.94  53 

.9004 

.6327 

.7364 

.6172 

.4744 

.3222 

.1798 

.0702 

.0113 

.9093 

.9767 

.9336 

.9140 

.0307 

.7360 

.6242 

.4537 

.2639 

.0861 

.9990 

.9973 

.9940 

.9663 

.9710 

.9437 

.0926 

.8031 

.6313 

.4013 

10 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.0005 

.0002 
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.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0039 

.0022 

.0007 

.0002 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0327 

.0140 

.0039 

.0020 

.0006 

.0001 

.0000 

.0000 

.0000 

.0000 

.1133 

.0610 

.0293 

.0122 

.0043 

.0012 

.0002 

.0000 

.0000 

.0000 

.2744 

.1730 

.0*94 

.0301 

.0216 

.0076 

.0020 

.0003 
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.0000 
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.3669 

.2443 

.1407 

.0702 

.0343 

.0117 

.0027 

.0003 
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.7236 

.6029 

.4672 

.3317 

.2103 

.1146 

.0504 

.0139 

.0020 

.0001 

.6067 

.0069 

.7037 

.3744 

.4304 

.2967 

.1611 

.0694 

.0165 
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.7*99 

.6073 

.5440 
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.0096 
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.9941 
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.6029 

.6779 
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10 
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1.0000 
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0 
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.2783 
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.3330 
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3 
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.9947 
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.7139 

.3744 
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6 
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.9999 

.9987 
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7 
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.9330 
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• 
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.9993 

.9973 
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.9997 

.9987 
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1.0000 

1.0000 

1.0000 

1.0000 

1.0000 
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13 
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1.0000 

1.0000 
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1.0000 

1.0000 

1.0000 
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1.0000 

0 
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.2288 

.1020 

.0440 

.0170 

.0068 
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1 
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.3846 

.3367 
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.1010 

.0473 

.0203 

.0081 

.0029 

2 

.9699 

.8416 

.6479 

.4481 

.2811 
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.0839 
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3 
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.3552 
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4 
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.1672 

3 
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.6403 

.4039 

.3373 

4 
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.9617 

.9067 
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7 
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.9997 
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.9998 
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it 

0 
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.0000 
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1 
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2 
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4 
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.0233 
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S 
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.2407 
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.0344 
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.0039 

.0007 
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.0000 

« 
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.3344 

.2127 

.1174 
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1 

.4042 

.4934 

.3414 
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.2743 

.1374 

.0724 

.0239 
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.0002 

4 

.9270 

.4435 

.77*7 

.6333 

.3073 

.3312 

.205* 

.0927 

.0234 

.0022 

4 

.9407 

.9379 

.9144 

.1*47 

.7*73 

.4093 

.4*17 

.34*2 

.1109 

.0194 

to 

.9944 

.9917 

.940* 

.9376 

.9130 

.4*16 

.  /  231 

.3345 

.3*10 

.114* 

n 

.9999 

.9992 

.9974 

.99*3 

.9842 

.4443 

.9313 

.4374 

.7174 
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12 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

13 

0 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

X 

.0017 

.0005 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0112 

.0041 

.0013 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0441 

.0203 

-.0074 

.0025 

.0007 

.0001 

.0000 

.0000 

.0000 

.0000 

4 

.1334 

.0494 

.0321 

.0126 

.0040 

.0010 

.0002 

.0000 

.0000 

.0000 

3 
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.1744 

.0977 

.0*62 

.0142 

.0034 

.0012 

.0002 

.0000 

.0000 

4 
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.3363 

.2241 

.2293 

.042* 

.02*3 

.0070 

.0013 

.0001 

.0000 

7 

.7093 

.3732 

.4234 

.24*1 

.143* 

.0402 

.0300 

.0033 

.0009 

.0000 

• 

.4446 

.7721 

.4*70 

.4993 

.3*37 

.2060 

.0991 

.0240 

.0043 

.0003 

* 

.9339 

.9071 

.431* 

.7217 

.574* 

.*137 

.2327 

.0947 

.03*2 

.0011 

10 

.9444 

.9731 

.9*21 

.4444 

.7473 

.447* 

.*441 

.270* 

.1339 

.02*3 

11 

.9943 

.9951 

.9*7* 

.970* 

.9343 

.4733 

.744* 

.4017 

.3747 

.113* 

12 

.9999 

.9994 

.9947 

.9963 

.9903 

.9762 

.9*30 

.4791 

.7*54 

.*447 

11 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

14 

0 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

1 

.0009 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0065 

.0022 

.0006 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0247 

.0114 

.0019 

.0011 

.0002 

.0000 

.0000 

.0000 

.0000 

.0000 

4 

.0494 

.0462 

.0173 

.0060 

.0017 

.0003 

.0000 

.0000 

.0000 

.0000 

3 

.2120 

.1149 

.0313 

.02*1 

.0043 

.0022 

.000* 

.0000 

.0000 

.0000 

4 

.3933 

.2316 

.1301 

.0733 

.0313 

.0103 

.002* 

.0003 

.0000 

.0000 

7 

.6047 

.4339 

.3075 

.1136 

.0933 

.0343 

.0114 

.0022 

.0002 

.0000 

4 

.7440 

.4427 

.31*1 

.3393 

.2193 

.1117 

.0*39 

.0113 

.0013 

.0000 

* 

.9102 

.4324 

.7207 

.3773 

.4134 

.2343 

.1294 

.0*47 

.0092 

.000* 

10 

.9713 

.9344 

.4737 

.7743 

.4**4 

.*747 

.3014 

•  1*63 

.0**1 

.00*2 

n 

.9933 

.9430 

.4402 

.9161 

.4392 

.7149 

.3319 

.3321 

.154* 

.0301 

12 

.9991 

.9971 

.9919 

.4743 

.9323 

.4490 

.1021 

.4*33 

.*13* 

.1310 

13 

.9999 

.9996 

.9992 

.4976 

.9932 

.9422 

.43*0 

.4972 

.7712 

.3123 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 
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f 
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.10 
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D 

D 
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.11 
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.41 

IS 

0 
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.1019 

.0174 

.Nil 

.0114 
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.0014 

.0001 
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i 
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.0011 

.0017 

| 
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.SIM 
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.1141 

.1IM 

.0417 

.0171 

.0M7 

) 

.9941 

.9444 

.1117 

.4411 

.4411 

.1949 

.1717 

.0901 
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s 

.9004 

.9971 

.9M) 
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.1111 

.1119 
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.1104 

f 

.9009 

.9971 

.Nil 

.9M9 

.1114 
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.1441 

.4011 
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« 
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•N19 

.0414 

.0449 
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• 
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•  9994 
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.9041 
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.7049 
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• 
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1.0000 

1.0000 

.9999 

.9991 

.994) 

.H74 

.9441 

.9111 

10 

1.0000 

1.0090 

1.0000 

1.0000 

.9999 

.9991 

.9971 

.9407 

.9741 

u 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9991 

.9941 

.99)7 

11 
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1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.4449 

.9N7 

.9949 

11 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

IS 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

IS 

0 

.4401 

.1111. 

.074} 

.0141 

.0100 

.001) 

.0010 

.000) 

.0001 

1 

.1100 

.1147 

.(•19 

.1407 

.0411 

.0141 

.0091 

.0011 

.0010 

1 

.9111 

.7191 

.1414 

.1111 

.1971 

.0994 

.0411 

.0101 

.0044 

1 

.9910 

.9111 

.7199 

.INI 

.4010 

.1419 

.1119 

.Mil 

.0141 

4 

.9991 

.9110 

.9109 

.7M1 

.4101 

.4499 

.1191 

.1444 

.041) 

S 

.9999 

.9947 

.9741 

.9141 

.110) 

.4191 

.4900 

.1144 

.1974 

s 

1.0000 

.9991 

.9944 

.9711 

.9104 

•  1147 

.4441 

.1171 

.1140 

7 
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.9999 

'*9919 

.9910 

.9719 

.9IM 

.4404 

.7141 

.1419 

• 

1.0000 

1.0000 

.9991 

.9991 

.Nil 

.9741 

.9119 

.4177 

.7441 

9 
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1,0000 

1.0000 

.9994 

.9944 

.9919 

.9771 

.9417 

.4719 
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1.0000 

1.0000 

1.0000 
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.9944 

.9910 

.4404 

.9114 
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.9991 
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1.0000 
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1.0000 

1.0000 

1.0000 
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1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9449 

IS 

1.0000 

1.0000 
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1.0000 
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IS 

1.0000 
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.0141 

.0036 

.0003 

.0000 

9 

.4491 

.7)92 
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.7439 

.4020 
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.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0021 

.0004 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0104 

.00)5 

.0009 

.0002 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

4 

.0)44 

.0149 

.0049 

.001) 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

S 

.10)1 

.0444 

.0191 

.0042 

.0014 

.0003 

.0000 

.0000 

.0000 

.0000 

4 

.2272 

.1241 

.0543 

.0229 

.0071 

.0014 

.0002 

.0000 

.0000 

.0000 

7 

.4014 

.2559 

.142) 

.0471 

.0257 

.007) 

.0013 

.0002 

.0000 

.0000 

• 

.S942 

.4)71 

.24)9 

.1)94 

.0744 

.0271 

.0070 

.0011 

.0001 

.0000 

9 

.7724 

.4)40 

.4724 

.3119 

.1753 

.0794 

.0267 

.0036 

.0003 

.0000 

10 
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12 
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.5930 

.4019 

.2101 

.0644 

.0070 

12 

.4974 

.99)4 

.9417 

.9)49 

.9004 

.4729 

.4442 

.4346 

.2104 

.0429 

14 

.4447 

.4440 

.9947 

.9902 

.97)9 

.9)43 

.4393 

.7141 

.44)3 

.1492 

IS 

1.0000 

.4444 

.9447 

.99  90 

.9947 

.9900 

.9719 

.9257 

.4147 

.3399 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

A-96 


Ttbla  II  (Conclnuad) 


n 

7 

p«.05 

.10 

.15 

.20 

.25 

.30 

.33 

.40 

.45 

17 

0 

.4181 

.1668 

.0631 

.0223 

.0075 

.0023 

.0007 

.0002 

.0000 

1 

.7922 

.4818 

.2525 

.1182 

.0301 

.0193 

.0067 

.0021 

.0006 

2 

.9497 

.7618 

.5198 

.3096 

.1637 

.0774 

.0327 

.0123 

.0041 

3 

.9912 

.9174 

.7356 

.3489 

.3530 

.2019 

.1028 

.0464 

.0184 

4 

.998$ 

.9779 

.9013 

.7582 

.5739 

.3887 

.2348 

.1260 

.0596 

3 

.9999 

.9933 

.9681 

.8943 

.7633 

.5968 

.4197 

.2639 

.1471 

6 

1.0000 

.9992 

.9917 

.9623 

.8929 

.7752 

.6188 

.4478 

.2902 

7 

1.0000 

.9999 

.9983 

.9891 

.9598 

.8954 

.7872 

.6403 

.4743 

a 

1.0000 

1.0000 

.9997 

.9974 

.9876 

.9397 

.9006 

.8011 

.6626 

a 

1.0000 

1.0000 

1.0000 

.9995 

.9969 

.9873 

.9617 

.9081 

.8166 

10 

1.0000 

1.0000 

1.0000 

.9999 

.9994 

.9968 

.9880 

.9652 

.9174 

n 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9993 

.9970 

.9894 

.9699 

12 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9994 

.9975 

.9914 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9995 

.9981 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9997 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

16 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

17 

1.0000 

1.0000  ■ 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

18 

0 

.3972 

.1301 

.0336 

.0180 

.0056 

.0016 

.0004 

.0001 

.0000 

1 

.7735 

.4303 

.2241 

.0991 

.0395 

.0142 

.0046 

.0013 

.0003 

2 

.9419 

.7338 

.4797 

.2713 

.1353 

.0600 

.0236 

.0082 

.0023 

3 

.9891 

.9018 

.7202 

.5010 

.3057 

.1646 

.0783 

.0328 

.0120 

4 

.9985 

.9718 

.8794 

.7164 

.5187 

.3327 

.1886 

.0942 

.0411 

3 

.9998 

.9936 

.'9581 

.8671 

.7175 

.5344 

.3350 

.2088 

.1077 

6 

1.0000 

.9988 

.9882 

.9487 

.8610 

.7217 

.5491 

.3743 

.2258 

7 

1.0000 

.9998 

.9973 

.9837 

.9431 

.8593 

.7283 

.5634 

.3915 

a 

1.0000 

1.0000 

.9993 

.9957 

.9807 

.9404 

.8609 

.7368 

.5778 

9 

1.0000 

1.0000 

.9999 

.*991 

.9946 

.9790 

.9403 

.8653 

,7473 

10 

1.0000 

1.0000 

1.0000 

.9998 

.9988 

.9939 

.9788 

.9424 

.8720 

11 

1.0000 

1.0000 

1.0000 

1.0000 

.9998 

.9986 

.9938 

.9797 

.9463 

12 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9997 

.9986 

.9942 

.9817 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9997 

.9987 

.9951 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1,0000 

1.0000 

.9998 

.9990 

— 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

16 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

17 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

ia 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

Tabla  II  (ConciQtiad) 


P-.50 

.33 

.60 

.65 

.70 

.75 

.80 

.83 

.90 

.95 

0 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

I 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0012 

.0003 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000  ' 

.0000 

3 

.0066 

.0019 

.0005 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

4 

.0263 

.0086 

.0025 

.0006 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0717 

.0301 

.0106 

.0030 

.0007 

.0001 

.0000 

.0000 

.0000 

.0000 

6 

.1662 

.0826 

.0348 

.0120 

.0032 

.0006 

.0001 

.0000 

.0000 

.0000 

7 

.3143 

.1834 

.0919 

.0383 

.0127 

.0031 

.0005 

.0000 

.0000 

.0000 

6 

.3000 

.3374 

.1989 

.0994 

.0403 

.0124 

.0026 

.0003 

.0000 

.0000 

9 

.6835 

.5257 

.3595 

.2128 

.1046 

.0402 

.0109 

.0017 

.0001 

.0000 

10 

.6338 

.7098 

.5522 

.3812 

.2248 

.1071 

.0377 

.0083 

.0008 

.0000 

11 

.9283 

.8529 

.7361 

.5803 

.4032 

.2347 

.1057 

.0319 

.0047 

.0001 

12 

.9755 

.9404 

.8740 

.7632 

.6113 

.4261 

.2418 

.0987 

'  .0221 

.0012 

13 

.9936 

.9816 

.9536 

.8972 

.7981 

.6470 

.4511 

.2444 

.0826 

.0088 

14 

.9988 

.9959 

.9877 

.9673 

.9226 

.8363 

.6904 

.4802 

.2382 

.0503 

IS 

.9999 

.9994 

.9979 

.9933 

.9807 

.9499 

.8818 

.7475 

.5182 

.2078 

16 

1.0000 

1.0000 

.9998 

.9993 

.9977 

.9925 

.9775 

.9369 

.8332 

.5819 

17 

1.0000 

1.0000 

r.oooo 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

0 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

1 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0007 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0038 

.0010 

.0002 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

4 

.0154 

.0049 

.0013 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0481 

.0183 

.0058 

.0014 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

6 

.1189 

.0537 

.0203 

.0062 

.0014 

.0002 

.0000 

.0000 

.0000 

.0000 

7 

.2403 

.1280 

.0576 

.0212 

.0061 

.0012 

.0002 

.0000 

.0000 

.0000 

• 

.4073 

.2527 

.1347 

.0597 

.0210 

.0054 

.0009 

.0001 

.0000 

.0000 

9 

.3927 

.4222 

.2632 

.1391 

.0596 

.0193 

.0043 

.0003 

.0000 

.0000 

10 

.7597 

.6085 

.4366 

.2717 

.1407 

.0569 

.0163 

.0027 

.0002 

.0000 

11 

.8811 

.7742 

.6237 

.4509 

.2783 

.1390 

.0513 

.0118 

.0012 

.0000 

12 

.9519 

.8923 

.7912 

.6450 

.4656 

.2823 

.1329 

.0419 

.0064 

.0302 

13 

.9846 

.9589 

.9058 

.8114 

.6673 

.4813 

.2836 

.1206 

.0282 

.0015 

14 

.9962 

.9880 

.9672 

.9217 

.8354 

.6943 

.4990 

.2798 

.0982 

.0109 

13 

.9993 

.9975 

.9918 

.9764 

.9400 

.8647 

.7287 

.5203 

.2662 

.0581 

16 

.9999 

.9997 

.9987 

.9954 

.9858 

.9605 

.9009 

.7759 

.5497 

.2265 

17 

1.0000 

1.0000 

.9999 

.9996 

.9984 

.9944 

.9820 

.9464 

.8499 

.6028 

ia 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

A-98 


Tabla  II  (Contlnuad) 


•»  y 


p“.05 

.10 

.15 

.20 

.25 

.30 

.33 

.40 

.45 

0 

.3774 

.1351 

.0456 

.0144 

.0042 

.0011 

.0003 

.0001 

.0000 

I 

.7347 

.4203 

.1985 

.0829 

.  jio 

.0104 

.0031 

.0008 

.0002 

2 

.9335 

.7054 

.4413 

.2369 

.1113 

.0462 

.0170 

.0055 

.0015 

3 

.9869 

.8650 

.6841 

.4551 

.2631 

.1332 

.0591 

.0230 

.0077 

4 

.9980 

.9648 

.8556 

.6733 

.4654 

.2822 

.1500 

.0696 

.0260 

5 

.9998 

.9914 

.9463 

.8369 

.6678 

.4739 

.2968 

.1629 

.0777 

« 

1.0000 

.9983 

.9837 

.9324 

.6251 

.6655 

.4812 

.3081 

.1727 

7 

1.0000 

.9997 

.9939 

.9767 

.9225 

.6180 

.6656 

.4876 

.3169 

$ 

1.0000 

1.0000 

.9992 

.9933 

.9713 

.9161 

.8145 

.6675 

.4940 

9 

1.0000 

1.0000 

.9999 

.9984 

.9911 

.9674 

.9125 

.8139 

.6710 

10 

1.0000 

1.0000 

1.0000 

.9997 

.9977 

.9895 

.9653 

.9115 

.6159 

U 

1.0000 

1.0000 

1.0000 

1.0000 

.9995 

.9972 

.9886 

.9648 

.9129 

12 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9994 

.9969 

.9884 

.9658 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9993 

.9969 

.9891 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9994 

.9972 

13 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

.9995 

16 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9999 

17 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

IS 

1.0000 

1.0000. 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

19 

1.0000 

1.0000  • 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

0 

.3365 

.1216 

.0388 

.0115 

.0032 

.0008 

.0002 

.0000 

.0000 

1 

.7358 

.3917 

.1756 

.0692 

.0243 

.0076 

.0021 

.0005 

.0001 

2 

.9245 

.6769 

.4049 

.2061 

.0913 

.0355 

.0121 

.0036 

.0009 

3 

.9841 

.8670 

.6477 

.4114 

.2252 

.1071 

.0444 

.0160 

.0049 

4 

.9974 

.9568 

.8298 

.6296 

.'4148 

*  .2375 

.1182 

.0510 

.0189 

5 

.9997 

.9887 

.9327 

.6042 

.6172 

.4164 

.2454 

.1256 

.0553 

6 

1.0000 

.9976 

.9781 

.9133 

“77858 

*  .6080 

.4166 

.2500 

.1299 

7 

•  1.0000 

.9996 

.9941 

.9679 

.8982 

.7723 

.6010 

.4159 

.2520 

8 

1.0000 

.9999 

.9987 

.9900 

.9591 

.8867 

.7624 

.5956 

.4143 

9 

1.0000 

1.0000 

.9998 

.9974 

.9861 

.9520 

.8782 

.7553 

.5914 

10 

1.0000 

1.0000 

1.0000 

.9994* 

.9961 

.9829 

.9468 

.8725 

.7507 

11 

1.0000 

1.0000 

1.0000 

.9999 

-  .9991 

.9949 

.9804 

.9435 

.6692 

12 

1.0000 

1.0000 

1.0000 

1.0000 

,  .9998 

.9987 

.9940 

.9790 

.9420 

13 

1.0000 

1.0000 

1.0000 

1.0000 

*  1.0000 

-  .9997 

.9985 

.9935 

.9786 

14 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

.9997 

.9984 

.9936 

13 

1.0000 

1.0000 

1.0000 

1.0000 

-2.0000 

1.0000 

1.0000 

.9997 

.9985 

16 

1.0000 

1.0000 

1.0000 

1.0000 

l.TJOOO 

1.0000 

1.0000 

1.0000 

.9997 

17 

1.0000 

1.0000 

1.0000 

1.0000 

••  i.oooo 

1.0000 

1.0000 

1.0000 

1.0000 

18 

1.0000 

1.0000 

1.0000 

1.0000 

iloooo 

1.0000 

1.0000 

1.0000 

1.0000 

_19 

..  1.0000 

1.0000 

1.0000“ 

-1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

20 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

A-99 


Table  II  (Continued) 


«  y 


p».S0 

.53 

.60 

.63 

.70 

.73 

.80 

.83 

.90 

.93 

0 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

1 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0004 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0022 

.0005 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000  - 

.0000 

A 

.0096 

.0028 

.0006 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

5 

.0318 

.0109 

.0031 

.0007 

.0001 

.0000 

.0000 

.0000 

.0000 

.0000 

6 

.0833 

.0342 

.0116 

.0031 

.0006 

.0001 

.0000 

.0000 

.0000 

.0000 

y 

.1796 

.0871 

.0332 

.0114 

.0028 

.0003 

.0000 

.0000 

.0000 

.0000 

B 

.3238 

.1841 

.0885 

.0347 

.0103 

.0023 

.0003 

.0000 

.0000 

.0000 

9 

.3000 

.3290 

.1861 

.0875 

.0326 

.0089 

.0016 

.0001 

.0000 

.0000 

10 

.6762 

.3060 

.3325 

.1853 

.0839 

.0287 

.0067 

.0008 

.0000 

.0000 

11 

.8204 

.6831 

.3122 

.3344 

.1820 

.0775 

.0233 

.0041 

.0003 

.0000 

12 

.9165 

.8273 

.6919 

.3188 

.3343 

.1749 

.0676 

.0163 

.0017 

.0000 

13 

.9682 

.9223 

.8371 

.7032 

.3261 

.3322 

.1631 

.0337 

.0086 

.0002 

14 

.9904 

.9720 

.9304 

.8300 

.7178 

.3346 

.3267 

.1444 

.0332 

.0020 

13 

.9978 

.9923 

.9770 

.9409 

.8668 

.7369 

.5449 

.3159 

.1150 

.0132 

16 

.9996 

.9985 

.9943 

.9830 

.9538 

.8887 

.7631 

.5587 

.2946 

.0663 

17 

1.0000 

.9998 

.9992 

.9969 

.9896 

.9690 

.9171 

.8013 

.5797 

.2453 

IB 

1.0000 

1.0000 

■i  9999 

.9997 

.9989 

.9938 

.9856 

.9344 

.8649 

.6226 

19 

1.0000 

i.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

0 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

1 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

2 

.0002 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

3 

.0013 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

4 

.0059 

.0015 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

5 

.0207 

.0064 

.0016 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

.0000 

6 

.0577 

.0214 

.0063 

.0013 

.0003 

.0000 

.0000 

.0000 

.0000 

.0000 

7 

.1316 

.0580 

.0210 

.0060 

.0013 

.0002 

.0000 

.0000 

.0000 

.0000 

8 

.2517 

.1308 

.0565 

.0196 

.0051 

.0009 

.0001 

.0000 

.0000 

.0000 

9 

.4119 

.2493 

.1275 

.0532 

.0171 

.0039 

.0006 

.0000 

.0000 

.0000 

10 

.3881 

.4086 

.2447 

.1218 

.0480 

.0139 

.0026 

.0002 

.0000 

.0000 

11 

.7483 

.5857 

.4044 

.2376 

.1133 

.0409 

.0100 

.0013 

.0001 

.0000 

12 

.8684 

.7480 

.5841 

.3990 

.2277 

.1018 

.0321 

.0050 

.0004 

.0000 

13 

.9423 

.8701 

.7300 

.3834 

.3920 

.2142 

.0867 

.0219 

.0024 

.0000 

14 

.9793 

.9447 

.8744 

.7546 

.3836 

.3828 

.1938 

.0673 

.0113 

.0003 

IS 

.9941 

.9811 

.9490 

.8818 

.7625 

.3852 

.3704 

.1702 

.0432 

.0026 

16 

.9987 

.9951 

.9840 

.9536 

.8929 

.7748 

.5886 

.3523 

.1333 

.0159 

17 

.9998 

.9991 

.9964 

.9879 

.9643 

.9087 

.7939 

.3951 

.3231 

.0755 

18 

1.0000 

.9999 

.9995 

.9979 

.9924 

.9757 

.9308 

.8244 

.6083 

.2642 

19 

1.0000 

1.0000 

1.0000 

.9998 

.9992 

.9968 

.9883 

.9612 

.8784 

.6415 

20 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

Tax  a  larger  than  20,  the  rth  quanta*  yt  of  a  binomial  random  variable  u;  bo  approximated 
ualng  yr-nir*vr/op<l-p) ,  vhora  vr  la  the  rth  quaaella  of  •  etandard  normal  ■/andon  variable, 
obtained  fro*  Table  AX. 
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Table  111  Quantile*  of  eh*  Lilllefors  Teat  Statistic  for  Nonullty 


pa.  80 

.83 

.90 

.95 

.99 

Saaple  else  n  ■  * 

.300 

.319 

.332 

.381 

.417 

3 

.283 

.299 

.313 

.337 

.403 

6 

.263 

.277 

.294 

.319 

.364 

7 

.247 

.238 

.276 

.300 

.348 

1 

.233 

.244 

.261 

.283 

.331 

9 

.223 

.233 

.249 

.271 

.311 

10 

.US 

.224 

.239 

.258 

.294 

11 

.206 

.217 

.230 

.249 

.284 

12 

.199 

,U2 

.223 

.242 

.273 

13 

.190 

.202 

.  U4 

.234 

v  .268 

14 

.183 

.194 

.207 

.227 

.261 

IS 

.177 

.187 

.201 

.220 

.237 

16 

.173 

.182 

.195 

.213 

•  .250 

17 

.169 

.177 

.189 

.206 

.245 

18 

.166 

.173 

.184 

.200 

.239 

19 

.163 

.169 

.179 

.195 

.235 

20 

.160 

.166 

.174 

.190 

.231 

23 

.142 

.147 

.158 

.173 

.200 

30  . 

.131 

.136 

.144 

.161 

.187 

Over  30 

.736 

.768 

.80S 

.886 

1.031 

A 

/S 

✓5 

/a 

/n 

SOURCE.  Adapted  fro*  Table  1  of 

Lilllefora 

(1967),  with 

correction*. 

fro* 

Conover  (10) .  - 
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